convez.ai
AgenteDevOps & InfraestruturaAvançado
alfredjose17

K8s Diagnostics Agent

alfredjose17/k8s-diagnostics-agent

Diagnóstico inteligente de erros K8s com confirmação human-in-the-loop

Ver no GitHub
InicialPluginScore · de 0 a 100

Agent de IA autônomo para diagnóstico e resolução de problemas em clusters Kubernetes via Telegram. Utiliza Claude para investigar falhas em tempo real, identificar causas raiz e executar correções após confirmação do usuário, com redação automática de dados sensíveis.

Instalação

$npx convez-ai alfredjose17/k8s-diagnostics-agent

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 6 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Troubleshooting automático de ImagePullBackOff, CrashLoopBackOff e outros erros K8s
  • Investigação de problemas desconhecidos via ReAct agentic loop
  • Escalação e restart de deployments com confirmação via Telegram

Veredito

Solução sólida e produção-ready para SREs que operam EKS. A arquitetura de dois estágios (fast path + full agent) é bem pensada e reduz custos de API. Requer familiaridade profunda com K8s, RBAC e AWS; não é ferramenta amadora.

README

A tradução para o português deste README ainda está na fila.

K8s Diagnostics Agent

An agentic Kubernetes troubleshooting assistant that lives inside your EKS cluster. Describe a problem in plain English via Telegram — the agent autonomously investigates your cluster using live data, diagnoses the root cause, and executes the fix if you confirm.

Built with the Anthropic Claude API's tool-use feature. No LangChain. No kubectl binary. No static credentials.


Architecture

Architecture

Telegram (user prompt)
        ↓
Bot handler (python-telegram-bot)
        ↓
Two-stage agent pipeline
  Stage 1 — Local triage (zero API tokens)
            Pattern match known errors → fetch targeted data
  Stage 2a — Focused Claude call (~800 tokens, known error)
  Stage 2b — Full ReAct loop (~3500 tokens, unknown error)
        ↓
Regex redaction layer (strips secrets, ARNs, tokens, IPs)
        ↓
Claude API (claude-sonnet-4-6) — diagnosis + fix command
        ↓
Telegram — "Apply this fix? yes/no"
        ↓
Kubernetes API (via ServiceAccount) — executes fix if confirmed

Features

  • Agentic investigation — Claude autonomously decides which tools to call based on what it observes, rather than following a hardcoded script
  • Two-stage pipeline — local pattern matching handles ~80% of common errors cheaply; full agent loop handles unknown issues
  • Human-in-the-loop — fix commands are never executed automatically; Telegram confirmation is always required
  • Data redaction — regex layer strips AWS credentials, tokens, connection strings, and internal IPs before any data reaches the Claude API
  • In-cluster deployment — runs as a Pod with a ServiceAccount; no kubeconfig, no static AWS credentials
  • Native Kubernetes API — uses the official Python client; no kubectl binary required in the container

Supported error types (fast path)

ErrorTools calledTokens
ImagePullBackOffget_pods + get_events~800
CrashLoopBackOffget_pods + get_logs~800
OOMKilledget_pods + describe_pod~800
Pendingget_pods + get_events~800
UnknownFull agent loop (up to 5 tools)~3500

Supported fix commands

The agent can propose and execute the following fix types after Telegram confirmation:

  • kubectl rollout restart deployment/<name>
  • kubectl scale deployment/<name> --replicas=<n>
  • kubectl delete pod/<name>

All other fixes are surfaced as manual instructions.


Project structure

k8s-diagnostics-agent/
├── agent/
│   ├── agent.py          # two-stage pipeline + ReAct agentic loop
│   ├── tools.py          # Kubernetes API wrappers
│   ├── redactor.py       # regex redaction layer
│   └── prompts.py        # Claude system prompt
├── bot/
│   ├── telegram_bot.py   # message handling + yes/no confirmation flow
│   └── state.py          # pending fix state store
├── k8s/
│   └── manifests/
│       ├── deployment.yaml
│       ├── serviceaccount.yaml
│       ├── rbac.yaml
│       └── secret.yaml       # not committed — see setup
├── .github/
│   └── workflows/
│       └── deploy.yml        # build → push → rollout on push to main
├── main.py
├── Dockerfile
└── requirements.txt

Prerequisites

  • EKS cluster (any version)
  • AWS CLI configured locally
  • kubectl installed locally
  • Telegram bot token (create via @BotFather)
  • Anthropic API key (console.anthropic.com)
  • Docker Hub account

Local setup

1. Clone and install dependencies

git clone https://github.com/<your-username>/k8s-diagnostics-agent.git
cd k8s-diagnostics-agent

python -m venv .venv
source .venv/bin/activate

pip install -e .

2. Configure environment variables

cp .env.example .env
# Edit .env and fill in your keys
# .env
ANTHROPIC_API_KEY=sk-ant-...
TELEGRAM_BOT_TOKEN=your-telegram-token

3. Connect kubectl to your EKS cluster

aws eks update-kubeconfig \
  --region <your-region> \
  --name <your-cluster-name>

4. Run locally

python main.py

Send your bot a message in Telegram:

are there any issues in kube-system?
check payment-service -n production
why are my pods crashing -n staging

Deploying to EKS

1. Create Kubernetes secrets

cp k8s/manifests/secret.yaml.example k8s/manifests/secret.yaml
# Edit secret.yaml and fill in your keys
kubectl apply -f k8s/manifests/secret.yaml

2. Apply RBAC and ServiceAccount

kubectl apply -f k8s/manifests/serviceaccount.yaml
kubectl apply -f k8s/manifests/rbac.yaml

3. Build and push Docker image

docker build -t <your-dockerhub-username>/k8s-diagnostics-agent:latest .
docker push <your-dockerhub-username>/k8s-diagnostics-agent:latest

4. Deploy

Update the image name in k8s/manifests/deployment.yaml, then:

kubectl apply -f k8s/manifests/deployment.yaml

5. Verify

kubectl get pods -l app=k8s-diagnostics-agent
kubectl logs -f deployment/k8s-diagnostics-agent

CI/CD (GitHub Actions)

Every push to main triggers:

  1. Build and push Docker image to Docker Hub (tagged with git SHA)
  2. Authenticate to AWS via OIDC (no static credentials stored)
  3. Update EKS deployment to the new image SHA
  4. Wait for rollout to complete

Required GitHub secrets:

SecretDescription
DOCKERHUB_USERNAMEDocker Hub username
DOCKERHUB_TOKENDocker Hub access token
AWS_ROLE_ARNIAM role ARN for OIDC auth
AWS_REGIONEKS cluster region
EKS_CLUSTER_NAMEEKS cluster name

See .github/workflows/deploy.yml for the full pipeline.


Security

In-cluster auth — the agent uses a Kubernetes ServiceAccount with a minimal ClusterRole. It can read pods, logs, and events, and write only the three supported fix operations. It cannot read Secrets or access other namespaces.

No static credentials — AWS authentication in CI uses GitHub Actions OIDC. The IAM role trust policy is scoped to the main branch only.

Data redaction — the following patterns are stripped before any data reaches the Claude API:

  • AWS credentials and ARNs
  • Bearer tokens and JWT strings
  • password=, token=, secret=, api_key= key-value pairs
  • Database connection strings with embedded credentials
  • Internal IP addresses (10.x, 172.16-31.x, 192.168.x)
  • Long base64-encoded strings (likely encoded secrets)

Secret values are never fetched — the tools.py implementation never calls kubectl get secret or reads raw Secret resource values. This is defence-in-depth on top of the redaction layer.


Example interaction

You:   payment-service pods keep crashing -n production

Bot:   🔍 Investigating your cluster...
       📋 Checking pod status...
       📜 Reading pod logs...

       ## Diagnosis
       The payment-service pod is in CrashLoopBackOff due to a
       database connection failure on startup.

       ## Root Cause
       The DB_HOST environment variable points to an unreachable
       endpoint. The pod crashes immediately with exit code 1.

       ## Fix
       Restart the deployment after verifying the DB_HOST value
       in the ConfigMap is correct.

       ## Prevention
       Add a readiness probe so the pod only receives traffic
       after a successful database connection.

       ⚠️ Suggested fix:
       kubectl rollout restart deployment/payment-service -n production

       Apply this to the cluster? Reply yes to confirm or no to cancel.

You:   yes

Bot:   ✅ Deployment 'payment-service' rollout restart triggered.

Tech stack

LayerTechnology
LanguagePython 3.11
LLMClaude (claude-sonnet-4-6) via Anthropic API
Agent patternReAct tool-use loop (no LangChain)
Kubernetes clientkubernetes Python SDK
Bot interfacepython-telegram-bot (async)
Container runtimeDocker
RegistryDocker Hub
CloudAWS EKS
CI/CDGitHub Actions + OIDC

Environment variables

VariableRequiredDescription
ANTHROPIC_API_KEYYesAnthropic API key
TELEGRAM_BOT_TOKENYesTelegram bot token from BotFather
K8S_NAMESPACENoDefault namespace (defaults to default)

License

MIT

Ver o original no GitHub

Bom para

Relacionados