convez.ai
MCP Server
Smife1023

self-healing-ops

Smife1023/self-healing-ops

Ver no GitHub
InicialPluginScore · de 0 a 100

Self-Healing AIOps system using MCP multi-agent collaboration. Simulates e-commerce microservice architecture with 8 production fault scenarios, automated root cause analysis, and intelligent repair with rollback support.

Instalação

$npx convez-ai Smife1023/self-healing-ops

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
1
Forks
0
Acompanhando
0
Saúde
28%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

README

A tradução para o português deste README ainda está na fila.

Self-Healing Ops v2.0

Intelligent AIOps Self-Healing System

MCP Multi-Agent based automated fault detection, diagnosis, and repair platform

Realistic e-commerce microservice architecture - 8 production-grade fault scenarios - 11 repair actions - End-to-end self-healing

Python FastAPI MCP LLM License


Architecture

Architecture

+-------------------------------------------------------------------------+
|                   E-Commerce Microservice Platform                        |
|                                                                          |
|   +----------+    +---------------------------+    +-----------------+  |
|   |  Nginx   |--->|    API Gateway (Kong)      |    |  db-master-01   |  |
|   |  (LB)    |    +---+--------+--------+---+    |  MySQL Primary   |  |
|   +----------+        |        |        |         |  (32C/128G/2TB)  |  |
|                 +-----+--+ +---+---+ +--+----+   +--------+--------+  |
|                 | user-  | |order- | |payment|            |            |
|                 |service | |service| |service|    +-------+--------+  |
|                 +---+----+ +---+---+ +---+---+    |  db-replica-01 |  |
|                     |         |         |         |  MySQL Replica  |  |
|                 +---+---------+---------+---+    +----------------+  |
|                 |    app-server-01/02        |                        |
|                 |    (16C/32G each)          |    +-----------------+  |
|                 +----------------------------+    | cache-queue-01  |  |
|                                                   | Redis + RabbitMQ|  |
|   +------------------+                            | (8C/64G)        |  |
|   | notification-svc | <-- RabbitMQ              +-----------------+  |
|   | inventory-svc    |                                                 |
|   | Elasticsearch    |                                                 |
|   +------------------+                                                 |
+-------------------------------------------------------------------------+
                                 |
                     +-----------+-----------+
                     v           v           v
              +------------+ +---------+ +------------+
              |  Monitor   | | Diag    | |  Repair    |
              |   Agent    | |  Agent  | |   Agent    |
              |  (Max Key) | | (Std 1) | |  (Std 2)   |
              |  :9101     | | :9102   | |  :9103     |
              +------------+ +---------+ +------------+
                     |           |           |
                     +-----------+-----------+
                                 v
                     +-----------------------+
                     |   MCP Coordinator     |
                     |   Pipeline Orchest.   |
                     |   Incident Timeline  |
                     |   Post-Mortem Report |
                     +-----------------------+

Features

Realistic Infrastructure

  • 6 servers: LB / App x2 / DB Master / DB Replica / Cache+Queue
  • 12 microservices: Full e-commerce chain (User -> Order -> Payment -> Inventory)
  • 17 dependency edges: gRPC / HTTP / MySQL / Redis / AMQP
  • Service dependency graph with cascading failure propagation

SRE Best Practices

  • SLO Burn-Rate: Google SRE 4-window alerting method
  • Escalation Policy: P0-P3 graduated auto-remediation
  • Blast Radius: Real-time fault impact assessment
  • Post-Mortem: Automatic incident timeline generation

3-Agent Collaboration

  • MonitorAgent (Tier-1): Fast detection, SLO analysis
  • DiagnosticAgent (Tier-2): Deep diagnosis, root cause localization
  • RepairAgent (Tier-3): Runbook execution, rollback protection
  • Separate API Keys: Simulates different service account tiers

Full Auto Self-Healing

  • 11 repair actions: restart / rollback / scale / index / circuit breaker
  • Per-step verification: Auto health check after each repair
  • Auto rollback: Immediate revert on failure
  • Side-effect reporting: Records impact of every operation

Quick Start

Requirements

DependencyVersion
Python3.10+
Conda envautodev-agent
LLM APIAny Anthropic-compatible API

Install and Run

# 1. Activate environment
conda activate autodev-agent

# 2. Enter project
cd projectidea/self-healing-ops

# 3. Install dependencies
pip install -r requirements.txt

# 4. Configure API Key (required for first run)
cp .env.example .env
# Edit .env file, fill in your API Key:
#   MONITOR_API_KEY=your-api-key
#   DIAGNOSTIC_API_KEY=your-api-key
#   REPAIR_API_KEY=your-api-key
#   LLM_MODEL=claude-3-5-sonnet-20241022

# 5. Run (single-process mode, recommended)
python main.py all high_cpu              # CPU spike
python main.py all memory_leak           # Memory leak
python main.py all service_crash         # Service crash
python main.py all db_slow               # DB slow query
python main.py all connection_pool_exhaustion  # Connection pool exhaustion
python main.py all disk_full             # Disk full
python main.py all cascading_failure     # Cascading failure
python main.py all deployment_regression # Deployment regression
python main.py all random               # Random scenario

# 6. Interactive selection
python main.py scenario

# 7. List all scenarios
python main.py scenarios

Distributed Mode

# Terminal 1: Start 3 Agent MCP Servers
python main.py agents

# Terminal 2: Run coordinator
python main.py run high_cpu

Fault Scenarios

#ScenarioLevelDescriptionRoot CauseKey Metrics
1high_cpuP0CPU spikeReDoS catastrophic backtrackingCPU 98.3%, Load 28.5
2memory_leakP0Memory leakJava Heap HashMap infinite growthMEM 96.8%, OOM x4, GC 4.5s
3service_crashP0Service crash + rollback failMissing deploy configexit 1, 503, CB Open
4db_slowP1DB slow queryFull table scan + row lock contention15.2s, lock wait 847
5connection_pool_exhaustionP1Connection pool exhaustionRedis connection leakmaxclients 10000
6disk_fullP0Disk fullBinlog unrotated for 90 daysDisk 98.5%
7cascading_failureP0Cascading failureRedis down -> cache stampede -> DB overload30% 5xx
8deployment_regressionP1Deployment regressionN+1 query bug847 queries/req

Self-Healing Pipeline

+-------------------------------------------------------------------------+
|                        Self-Healing Pipeline                              |
|                                                                          |
|  Step 1 ---> Step 2 ---> Step 3 ---> Step 4 ---> Step 5                |
|   Detect      Diagnose    Plan      Execute     Verify                  |
+-------------------------------------------------------------------------+
StageAgentCapabilitiesOutput
Step 1: DetectMonitorAgentSLO burn-rate, anomaly detection, cascading failure IDAlert JSON
Step 2: DiagnoseDiagnosticAgentTimeline rebuild, dependency BFS, blast radiusRoot cause report
Step 3: PlanRepairAgent4-phase Runbook, risk assessment, rollback planRepair plan JSON
Step 4: ExecuteRepairAgentStep-by-step exec, health check, rollback, side-effectsExecution result
Step 5: VerifyCoordinatorHealth check, SLO validation, timeline, Post-MortemFinal report

MCP Tools

MonitorAgent (:9101) - Tier-1 Detection

ToolDescription
monitor_checkFull monitoring + SLO analysis
get_infra_snapshotRaw monitoring data snapshot
check_service_healthSingle service health check
get_dependency_statusDependency status check

DiagnosticAgent (:9102) - Tier-2 Diagnosis

ToolDescription
diagnoseDeep root cause analysis
get_detailed_logsService detailed logs
get_dependency_graphFull dependency graph
trace_impactBFS fault propagation trace

RepairAgent (:9103) - Tier-3 Repair

ToolDescription
create_repair_planRepair Runbook generation
execute_repairExecute single repair step
verify_healthPost-repair health verification
rollback_last_actionRollback last operation

Repair Actions

ActionDescriptionRiskReversibleUse Case
restart_serviceGraceful service restartLowYesService crash/hang
rollback_deployRollback to previous versionMediumYesDeployment regression
kill_processSIGKILL specific processLowN/AProcess stuck
clear_cacheFLUSHALL RedisMediumNoConnection leak/cache pollution
scale_upHPA scale replicasLowYesTraffic surge
cleanup_diskClean logs/BinlogLowNoDisk full
fix_configInject config from VaultMediumYesMissing config
add_indexCreate database indexLowYesSlow query
failover_replicaPromote replica to primaryHighComplexPrimary unavailable
drain_connectionsDrain stale connectionsLowN/AConnection leak
circuit_breaker_resetReset circuit breakerLowAutoCB false trigger

Monitoring Metrics

Server Level (node_exporter)

  • CPU usage / cores
  • Memory usage / Swap
  • Disk usage / I/O Wait
  • Network RX/TX (Mbps)
  • Load Average (1m/5m/15m)
  • TCP connections / TIME_WAIT
  • Open Files / Uptime

Application Level (application_metrics)

  • Response time p50 / p99
  • Error rate (5xx/4xx)
  • QPS (requests per minute)
  • Restart count
  • Connection pool (size/active)
  • Heap usage / GC pause
  • Circuit breaker state (closed/open/half-open)

Project Structure

self-healing-ops/
|-- config.py              # API Key, thresholds, SLA, escalation policy
|-- infrastructure.py      # Infrastructure simulator
|                            6 servers / 12 services / 17 deps / 8 faults
|-- mcp_agent_server.py    # MCP Server base (FastAPI + LLM + retry + metrics)
|-- coordinator.py         # Coordinator (pipeline + timeline + Post-Mortem)
|-- main.py                # Entry point (single-process / distributed / interactive)
|-- requirements.txt       # Python dependencies
|-- .env.example           # Environment variable template
|-- .gitignore             # Git ignore rules
|-- README.md              # This file
|
|-- agents/
|   |-- __init__.py
|   |-- monitor_agent.py   # Monitor Agent - SLO burn-rate + detection
|   |-- diagnostic_agent.py # Diagnostic Agent - dependency graph + root cause
|   |-- repair_agent.py    # Repair Agent - Runbook + rollback + verification

Configuration

API keys are configured via environment variables or .env file (do NOT hardcode API keys):

# .env file
MONITOR_API_KEY=sk-ant-xxx      # Tier-1 Monitor Agent
DIAGNOSTIC_API_KEY=sk-ant-xxx   # Tier-2 Diagnostic Agent
REPAIR_API_KEY=sk-ant-xxx       # Tier-3 Repair Agent
LLM_MODEL=claude-3-5-sonnet-20241022
MIMO_BASE_URL=https://api.anthropic.com

All 3 agents can use the same API key, or configure separately to simulate different quota tiers.

THRESHOLDS = {
    "cpu_critical": 90.0,              # CPU > 90% => CRITICAL
    "memory_critical": 92.0,           # MEM > 92% => CRITICAL
    "disk_critical": 90.0,             # DISK > 90% => CRITICAL
    "error_rate_critical": 0.10,       # Error rate > 10% => CRITICAL
    "latency_p99_critical_ms": 2000,   # p99 > 2s => CRITICAL
    "sla_target": 99.9,                # Three nines
    "error_budget_burn_rate_threshold": 14.4,  # 2% budget in 1h
}
ESCALATION_POLICY = {
    "P0_critical": {"auto_remediate": True,  "requires_approval": False},
    "P1_high":     {"auto_remediate": True,  "requires_approval": False},
    "P2_medium":   {"auto_remediate": True,  "requires_approval": True},
    "P3_low":      {"auto_remediate": False, "requires_approval": True},
}

v1.0 vs v2.0

Dimensionv1.0v2.0
Servers36 (LB / App x2 / DB x2 / Cache)
Services612 (Full e-commerce microservices)
Fault Scenarios68 (+cascading failure, +deployment regression)
Dependency GraphN/A17 edges + BFS traversal
MonitoringCPU/MemorySLO burn-rate / p99 / conn pool / GC / CB
DiagnosisSimple alertsDep graph traversal, timeline, blast radius
Repair Actions611 (+rollback/index/CB/conn drain)
RollbackN/APer-step rollback support
Event TrackingN/AFull timeline + Post-Mortem
Agent Tools612 (4 tools per agent)

Use Cases

  • MCP Multi-Agent collaboration demo
  • Operations automation demonstration
  • SRE best practices showcase
  • Fault diagnosis reasoning chain demo
  • Technical interview project showcase

Tech Stack

ComponentTechnology
Agent FrameworkCustom MCP Server (FastAPI)
LLMAny Anthropic-compatible API
CommunicationMCP (Model Context Protocol) over HTTP
InfrastructureSimulator (6 servers + 12 services + dep graph)
CoordinatorAsync HTTP orchestration + incident timeline
MonitoringSLO burn-rate, p50/p99, conn pool, GC, CB

Built by Self-Healing Ops Team

Ver o original no GitHub

Bom para

Relacionados