convez.ai
MCP Server
dumbL4d

sentinel-sre-agent

dumbL4d/sentinel-sre-agent

Ver no GitHub
InicialPluginScore · de 0 a 100

Self-improving SRE agent for ML model observability with Gemini, Phoenix tracing, and MCP

Instalação

$npx convez-ai dumbL4d/sentinel-sre-agent

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
1
Forks
1
Acompanhando
0
Saúde
42%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

README

A tradução para o português deste README ainda está na fila.

Sentinel SRE Agent

MIT License Python 3.11+ Live Demo

Self-improving SRE agent for ML model observability and incident response.

Built for the Google Cloud Rapid Agent Hackathon — Arize Partner Track.

What It Does

Sentinel is an AI agent that investigates ML production incidents by querying real-time metrics, traces, and drift signals — and gets smarter over time by querying its own past investigation traces via Arize Phoenix MCP, adapting its reasoning based on what worked before. Every investigation is traced to Phoenix Cloud, scored by an LLM judge, and the scores feed back into the agent's self-improvement loop for the next incident.

Try It in 60 Seconds

curl -X POST CLOUD_RUN_URL/run \
  -H "Content-Type: application/json" \
  -d '{"mission": "URGENT: Error rate on fraud-detection-v1 just spiked to 15%. Investigate immediately and tell me what to do."}'

Expected output (truncated):

{
  "content": "**Summary**\nFraud-detection-v1 is experiencing a critical error rate...\n\n**Evidence**\n...",
  "session_id": "...",
  "evaluation": {
    "overall_score": 4.5,
    "accuracy": 4.5,
    "completeness": 4.0,
    "actionability": 5.0,
    ...
  }
}

How the Self-Improvement Loop Works

sequenceDiagram
    participant Incident
    participant Agent
    participant PhoenixMCP as Phoenix MCP
    participant PhoenixCloud as Phoenix Cloud
    participant Judge as LLM Judge

    Incident->>Agent: New investigation
    Agent->>PhoenixMCP: self_introspect()
    PhoenixMCP->>PhoenixCloud: Query past traces + eval scores
    PhoenixCloud-->>PhoenixMCP: Similar past cases, scores, annotations
    PhoenixMCP-->>Agent: Historical context + insights
    Agent->>Agent: Adapt strategy based on what worked before
    Agent->>PhoenixCloud: Run investigation (traced via OpenInference)
    Agent-->>Incident: Investigation report
    Judge->>Judge: Score response (accuracy, completeness, actionability)
    Judge->>PhoenixCloud: Store evaluation scores
    Note over Agent,PhoenixCloud: Next incident → loop repeats with improved context

Architecture

flowchart TD
    User["User<br/>(CLI / Web UI)"]
    ADK["Vertex AI ADK Runner<br/>google-adk Agent"]

    subgraph Tools["11 Sentinel Tools"]
        SI["self_introspect<br/>→ Phoenix MCP"]
        QM["QueryMetrics"]
        QT["QueryTraces"]
        GA["GetAlerts"]
        AD["AnalyzeDrift"]
        CS["CorrelateSignals"]
        SR["SuggestRemediation"]
        CA["CreateAlert"]
        PT["QueryPhoenixTraces"]
        PS["QueryPhoenixSpans"]
        PSE["QueryPhoenixSessions"]
    end

    subgraph Phoenix["Arize Phoenix Cloud"]
        OTEL["OpenInference Tracing"]
        TRACES["Traces (spans)"]
        EVALS["Evaluations (LLM Judge)"]
        MCP["Phoenix MCP Server"]
    end

    Judge["LLM Judge (Gemini)<br/>→ Evaluation scores"]

    User --> ADK
    ADK --> Tools
    SI --> MCP
    MCP --> SI
    ADK --> OTEL
    OTEL --> TRACES
    TRACES --> MCP
    Judge --> EVALS
    EVALS --> MCP

Judging Criteria

CriterionWhat We BuiltWhere to See It
Uses Google Cloud Agent BuilderVertex AI ADK (google-adk 2.x) Agent + Runner + FunctionTool wrapping all 11 toolssrc/sentinel/agent/adk_agent.py
Arize Phoenix IntegrationOpenInference traces every LLM call to Phoenix Cloud; agent queries Phoenix via MCP for self-improvementsrc/sentinel/tracing.py, src/sentinel/mcp/phoenix_client.py
Self-Improvement Loopself_introspect tool queries Phoenix MCP for similar past cases and eval scores; agent adapts reasoning from historical patternssrc/sentinel/tools/self_introspect.py, system prompt in src/sentinel/agent/prompts.py
Multi-Tool Investigation11 tools: metrics, traces, alerts, drift analysis, signal correlation, alert creation, remediation suggestions, Phoenix introspectionsrc/sentinel/tools/ directory
LLM-as-a-Judge EvaluationGemini evaluates every response on accuracy, completeness, actionability (1-5 scale)src/sentinel/evaluation/llm_judge.py
Streaming Web UIVanilla JS dashboard with SSE streaming, health badges, phoenix/data-source indicatorssrc/sentinel/static/index.html, src/sentinel/server.py
Cloud Run DeployableDockerfile with healthcheck, GitHub Actions CI → Artifact Registry → Cloud RunDockerfile, .github/workflows/deploy.yml

Tech Stack

ComponentTechnology
Agent FrameworkVertex AI ADK (google-adk 2.x)
LLMGemini 2.5 Flash Lite
ObservabilityArize Phoenix Cloud (traces, spans, evaluations)
MCP Integration@arizeai/phoenix-mcp over stdio
TracingOpenInference + OpenTelemetry OTLP
EvaluationGemini LLM-as-a-Judge
Web UIVanilla JS, SSE (no build step)
API ServerFastAPI + Uvicorn
DeploymentCloud Run + GitHub Actions

Project Structure

sentinel-sre-agent/
├── src/sentinel/
│   ├── agent/
│   │   ├── adk_agent.py      # Vertex AI ADK agent (primary)
│   │   ├── core.py           # Legacy google-genai agent (fallback)
│   │   └── prompts.py        # System prompt with self-improvement guidance
│   ├── mcp/
│   │   ├── phoenix_client.py # Phoenix MCP client (real MCP protocol)
│   │   └── arize_client.py   # High-level Arize data access layer
│   ├── tools/
│   │   ├── query.py          # query_metrics (with Phoenix fallback), query_traces, get_alerts
│   │   ├── analyze.py        # analyze_drift, correlate_signals
│   │   ├── actions.py        # create_alert, suggest_remediation
│   │   ├── self_introspect.py# Self-improvement loop (queries Phoenix MCP)
│   │   └── phoenix_tools.py  # Raw Phoenix trace/span/session queries
│   ├── evaluation/
│   │   └── llm_judge.py      # LLM-as-a-judge evaluation
│   ├── static/
│   │   └── index.html        # Streaming web UI
│   ├── server.py             # FastAPI server with SSE + health
│   ├── tracing.py            # OpenInference → Phoenix tracing
│   ├── scenarios.py          # Demo scenarios
│   └── cli.py                # CLI entry point
├── Dockerfile
├── pyproject.toml
└── README.md

Running Locally

1. Clone and install:

git clone https://github.com/dumbL4d/sentinel-sre-agent.git
cd sentinel-sre-agent
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

2. Set environment variables (all required):

export GEMINI_API_KEY="your-key-from-aistudio.google.com"
export PHOENIX_API_KEY="your-key-from-app.phoenix.arize.com"
export PHOENIX_COLLECTOR_ENDPOINT="https://app.phoenix.arize.com"

3. Run the server:

sentinel-serve

4. Open the web UI:

http://localhost:8000/ui/

5. Or use the CLI:

sentinel interactive --demo

Known Limitations

  • QueryMetrics synthetic fallback: When a Phoenix project has no matching model spans (or returns < 5 spans), QueryMetrics silently falls back to seeded synthetic data. The data_source field in the response indicates whether data came from Phoenix or the synthetic generator.
  • Text-based similarity for self-improvement: The self-improvement loop finds past cases using token overlap (bag-of-words), not semantic embeddings. A future upgrade would use Phoenix's native embedding search for more relevant historical context retrieval.
  • No human-in-the-loop for remediation: create_alert and suggest_remediation execute immediately without requiring manual approval. A production deployment should gate remediation actions behind a human review step.

License

MIT

Ver o original no GitHub

Bom para

Relacionados