convez.ai
AgenteDevOps & InfraestruturaAvançado
mairp

Agent Observability Stack

mairp/agent-observability-stack

Observabilidade completa para agentes LLM + infraestrutura com onboarding automático

Ver no GitHub
InicialPluginScore · de 0 a 100

Stack de observabilidade auto-hospedável para agentes LLM e hosts Linux, com suporte nativo a Intel iGPU/NPU. Integra Prometheus, Grafana, Tempo e OpenTelemetry para coletar métricas de infraestrutura, traces de agentes, custos de tokens LLM e diagnósticos em tempo real.

Instalação

$npx convez-ai mairp/agent-observability-stack

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
1
Forks
0
Acompanhando
0
Saúde
57%
Último commit
há 1 mês
Versão
—
Issues abertas
1
Contribuidores
2

Casos de uso

  • Monitorar custos e latência de chamadas LLM em produção
  • Rastrear performance de agentes Claude Code com traces distribuídas
  • Coletar métricas de aceleração Intel (iGPU/NPU) e containers automaticamente
  • Alertas Telegram para anomalias em agentes e infraestrutura

Veredito

Stack robusto e pronto para produção, ideal para quem executa agentes LLM em infraestrutura própria e precisa visibilidade completa de custos, performance e saúde dos agentes. Exige familiaridade com Docker, Prometheus e observabilidade distribuída.

README

A tradução para o português deste README ainda está na fila.

agent-observability-stack

A self-hostable observability stack for LLM agents + a Linux host (Intel iGPU/NPU aware), built as a single LAN-accessible pane of glass. Two pipelines, one Grafana:

  • Infra metrics (pull): CPU, RAM, disk, network + latency, containers, and Intel iGPU/NPU telemetry → Prometheus.
  • Agent telemetry (push): OpenClaw diagnostics → OTLP → OpenTelemetry Collector → Grafana Tempo (traces) + Prometheus (metrics); LLM tokens, cost, latency, cache from a LiteLLM proxy.

New agents, containers, and accelerators onboard themselves — no per-target config edits.

Demo

Full walkthrough (~45s) — the dashboards end to end:

https://github.com/user-attachments/assets/dcd5292b-c909-4bfe-86aa-102e022481ee

Per-pipeline highlights:

Infra pipeline — host, accelerators, containers:

infra pipeline

Agents pipeline — cost, tokens, latency, call outcomes, traces, live agent inventory, plus the Claude Code & RAG dashboard (Claude Code token usage / cache / cost, and QMD index health):

agents pipeline

Architecture

INFRA METRICS (pull)                    AGENT / AI TELEMETRY (push)
 node_exporter ─┐                        OpenClaw gateway          Claude Code (OTel)
 cadvisor      ─┤                         ├─ diag-prometheus ─(loopback textfile)─┐   │
 blackbox      ─┼─► Prometheus ◄──────────┤                                       │   │ OTLP
 intel npu/gpu ─┤        ▲                 └─ diag-otel ──OTLP─┐                   │   ▼
 litellm /metrics┘       │                                    ▼                   └► OTel Collector
                         │                              OTel Collector ──traces──► Tempo
                         │                                   │  │  └────logs───────► Loki
                         └───────────────(metrics)───────────┘  │                     │
                      Grafana ◄────────── Prometheus + Tempo + Loki datasources ◄──────┘
                   (LAN :3000)   folders: Infrastructure · AI & Agents
                         ▲
        grafy-bot (Telegram /graph) · grafana-image-renderer (PNG) · Alertmanager → Telegram

Quickstart

cp .env.example .env          # fill in Grafana password, OpenClaw scrape token, Telegram bot, etc.
make up                       # render secrets + docker compose up -d
# host-side accelerator + diagnostics collectors (systemd):
sudo cp systemd/*.service systemd/*.timer /etc/systemd/system/
sudo systemctl enable --now accel-textfile.timer observability-onboard.timer

Open Grafana at http://<HOST_LAN_IP>:3000. Dashboards are provisioned into two folders: Infrastructure (host, accelerators, network, containers) and AI & Agents (Claude Code, LLM Cost & Consumption, OpenClaw Agents, RAG).

ServicePortPurpose
Grafana3000dashboards (LAN)
Prometheus9090metrics TSDB
Tempo3200trace store
Loki3100logs/events store (Claude Code per-request events)
OTel Collector4317/4318OTLP ingest — traces→Tempo, metrics→Prom, logs→Loki
Alertmanager9093alert routing → Telegram
grafana-image-renderer8081server-side PNG rendering (for the bot)
grafy-bot—Telegram /graph bot (image + values)

Claude Code metrics

Claude Code ships native OpenTelemetry. Enable it in your shell environment — add this to ~/.bashrc (or ~/.zshrc) and restart the session:

export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_METRIC_EXPORT_INTERVAL=10000
export OTEL_LOGS_EXPORT_INTERVAL=5000
export OTEL_RESOURCE_ATTRIBUTES=service.name=claude-code

Gotcha: putting these in ~/.claude/settings.json's env block does not work — that block isn't applied to Claude Code's telemetry exporter, so metrics stay at 0. It must be in the shell env.

The Claude Code dashboard then breaks consumption out by model (Opus/Sonnet/Haiku), by token type (input / output / cacheRead / cacheCreation), and by named subagent (Explore/Plan/… from subagent_completed events), plus cost, cache-read per model, request count and p95 latency. It is built on the per-request api_request events in Loki ({service_name="claude-code"}) rather than the OTLP counters — the events persist (the counters are ephemeral and short sessions exit before the metric flush). No prompt/response content is exported — counts and event metadata only. See docs/agents.md and docs/dashboards.md.

What you get

  • Host & virtualization-host dashboards (node_exporter), Intel Arc iGPU (busy %, freq, power) and Intel NPU (busy, freq, memory) panels, network reachability + latency (blackbox), per-container resources (cAdvisor).
  • Agents: per-model call latency (p50/p95), call volume & outcomes (OpenClaw), tokens + spend + prompt-cache hit ratio (LiteLLM), and per-run traces in Tempo. A live agent inventory table updates as agents are added.
  • Claude Code (by model + subagent): persistent consumption from per-request Loki events — cost & tokens per model (Opus/Sonnet/Haiku), tokens by type (input / output / cache-read / cache-creation), cache-read per model, request count & p95 latency, and named subagents (runs/tokens by agent_type — Explore/Plan/… — and by model), with a drill-down logs panel.
  • LLM Cost & Consumption: unified spend/tokens across Claude Code + OpenClaw/LiteLLM, by model/provider.
  • Memory breakdown: system RAM %, and how much is the iGPU / NPU (they share system RAM) — a "where's my RAM" view in bytes and % of total.
  • RAG: QMD index health (docs, vectors, freshness) per index.
  • Logs (Loki): the OTel Collector's logs pipeline ships Claude Code events to Loki; queryable in Grafana. Dashboards are grouped into Infrastructure and AI & Agents folders.
  • Alerting: label-matched rules (host pressure, target-down, NPU/iGPU saturation, model latency, daily spend budget) → Alertmanager → Telegram.
  • Telegram bot (Grafy): text /graph <dashboard> [range] to get a rendered dashboard image + key values on your phone, plus /values, /alerts, /list — chat-locked to you. See docs/telegram-bot.md.
  • Self-onboarding: a systemd timer + an optional OpenClaw observ cron-agent keep everything in sync as the fleet grows.

Docs

Tests

make validate   # static: compose, promtool (+ rule unit tests), amtool, otel, dashboards, shellcheck
make smoke      # live: targets up, metrics present, datasources healthy, Tempo trace ingest
make scan       # secret + host-leak scan

CI runs validate + a secret scan on every PR (.github/workflows/ci.yml).

License

MIT

Ver o original no GitHub

Bom para

Relacionados