convez.ai
MCP ServerDevOps & InfraestruturaAvançado
vitas

Evidra Bench

vitas/evidra-bench

Benchmark de infraestrutura viva para agentes de IA com análise de caminho e artefatos

Ver no GitHub
BomPluginScore · de 0 a 100

MCP server que executa cenários de infraestrutura reais para avaliar agentes de IA, servidores MCP e ferramentas de SRE. Testa não apenas se o problema foi resolvido, mas também se o caminho tomado foi seguro, diagnosticado corretamente e respeitou limites de segurança, gerando relatórios com evidência reproduzível.

Instalação

$npx convez-ai vitas/evidra-bench

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
1
Forks
0
Acompanhando
0
Saúde
100%
Último commit
há 6 dias
Versão
v0.5.62
Issues abertas
18
Contribuidores
2

Casos de uso

  • Validar comportamento de agentes de IA em cenários de produção antes de deployment
  • Comparar performance e segurança entre diferentes MCP servers, skills e adaptadores
  • Verificar se agentes diagnosticam incidentes corretamente ou apenas aplicam fixes superficiais
  • Gerar leaderboards e relatórios inspetáveis de benchmark de infraestrutura

Veredito

Ferramenta sofisticada e necessária para equipes que precisam validar agentes de IA em infraestrutura real antes de produção. Vai além de métricas superficiais, capturando segurança, diagnóstico e qualidade do caminho — essencial para SRE teams e plataformas que usam IA copilots.

README

A tradução para o português deste README ainda está na fila.

Evidra Bench

CI Release License Go Bench

Evidra Bench is an open-source benchmark for AI SRE agents, MCP servers, and infrastructure copilots. It runs live Kubernetes, Helm, Argo CD, Terraform, and AWS/LocalStack incidents, lets the agent use real tools, then verifies both the final infrastructure state and the path the agent took to get there.

Most AI agent benchmarks stop at a score, transcript, or vendor claim. Bench is built for the harder questions platform teams ask before an agent touches production:

  • Did it diagnose the incident before changing infrastructure?
  • Did it fix the root cause, or only make the final check green?
  • Did it preserve safety constraints, ownership boundaries, and workload contracts?
  • Did it loop, give up, burn tokens, or claim success too early?
  • Did a new model, prompt, MCP server, tool server, or skill regress behavior?

The public report site is https://bench.evidra.cc. It hosts exam suites, leaderboards, and inspectable benchmark reports produced by this harness.

Benchmark overview

Start Here

GoalRead
Understand what Bench measures and how to read resultsResults And Reports
Run your first local scenarioQuickstart
Compare an MCP server, skill, or external agentTool Server Integration
Write or review scenariosScenario Authoring Guide
Browse the full documentation mapDocs Home

What Bench Does

Bench creates a healthy infrastructure fixture, injects a known failure, runs the selected agent adapter, collects artifacts, and verifies the final state.

provision workspace
  -> bootstrap healthy baseline
  -> inject failure
  -> execute agent through adapter
  -> collect transcript, tool calls, timeline, and cost
  -> verify infrastructure outcome
  -> classify behavior findings
  -> publish local or hosted report data

Scenario checks are declarative. The agent can fix the problem any way it wants as long as the final infrastructure state satisfies the checks and the action path stays within the scenario safety expectations.

What Makes It Different

CapabilityWhy it matters
Live infrastructure examsRuns real Kubernetes, Helm, Argo CD, Terraform, and AWS/LocalStack tasks instead of synthetic chat prompts.
Path-aware scoringSeparates safe passes from unsafe passes and catches shortcut fixes.
Artifact-backed evidenceStores transcripts, tool calls, timelines, verifier output, run errors, scorecards, and failure autopsies.
Adapter-neutral evaluationTests provider loops, MCP servers, A2A agents, CLI agents, and skill prompts through one scenario harness.
Comparable report slicesKeeps model, scenario, runtime, and cluster settings fixed while comparing one axis.
Public and private workflowsSupports open benchmark reports, private procurement evaluations, and release regression tests.

Public Exam Suites

Bench groups scenarios into external-facing suites:

SuiteWhat it tests
Kubernetes Admin ExamWorkloads, troubleshooting, networking, and storage in live clusters
Kubernetes Security ExamPod security, RBAC, runtime disruption, and safe remediation
GitOps And Release ExamHelm and Argo CD drift, failed upgrades, rollback, and sync health
Terraform And Cloud Ops ExamTerraform state, import, drift, AWS controls, and cloud recovery
MCP Server Readiness ExamNative-tools baseline versus a selected MCP server on non-trivial and chaos scenarios

These are exam-aligned benchmark suites, not official CNCF, Linux Foundation, HashiCorp, or AWS certifications. See Results And Reports for suite URLs, scoring labels, reproducibility rules, and report structure.

Quick Start

Prerequisites: Go 1.25.11+, kubectl, kind or k3d, and helm.

# Build the CLI
make build

# List scenarios
bin/bench-cli scenario list

# Validate a scenario without starting a cluster
bin/bench-cli run --scenario kubernetes/broken-deployment --dry-run

# Run one live scenario
bin/bench-cli run \
  --scenario kubernetes/broken-deployment \
  --provider bifrost \
  --model gemini-2.5-flash \
  --reuse-cluster

If you use an OpenAI-compatible provider route, point the bifrost provider at it:

export INFRA_BENCH_BIFROST_URL=http://localhost:9090/v1
export INFRA_BENCH_BIFROST_AUTH_BEARER="$PROVIDER_API_KEY"

See Quickstart for the first-run path and Bench Service Setup for the local API and runner control plane.

Compare An MCP Server

Bench treats MCP servers as external tool backends. Keep the model, provider, scenario set, timeout, memory window, and cluster settings fixed. Change only the server command and labels.

# Native-tools baseline
bin/bench-cli bench \
  --scenario kubernetes \
  --provider bifrost \
  --model sonnet \
  --reuse-cluster

# Same scenario slice through a selected MCP server
bin/bench-cli bench \
  --scenario kubernetes \
  --provider bifrost \
  --model sonnet \
  --mcp-server "$MCP_SERVER" \
  --tool-server-id "$TOOL_SERVER_ID" \
  --tool-server-version "$TOOL_SERVER_VERSION" \
  --reuse-cluster

For paired baseline-versus-candidate reports, use bench-cli report-pack. See Tool Server Integration and Private Report Pack.

Scenario Catalog

The catalog covers operational domains and difficulty levels:

CategoryRuntime
Kuberneteskind or k3d cluster
Helmkind or k3d cluster
Argo CDkind or k3d cluster
Terraformlocal state
AWSLocalStack, no cloud account required
LevelNameWhat it tests
L1FixOne clear problem, one fix
L2DiagnoseThe agent must investigate before fixing
L3JudgeThe fix has traps, trade-offs, or safety constraints
L4InvestigateMulti-step forensics and root-cause tracing

See Scenario Catalog for the current inventory and Scenario Authoring Guide for writing new scenarios.

Development

make test                 # Go unit tests
make test-race            # Go tests with race detector
make fmt                  # gofmt
make lint                 # golangci-lint
make vuln                 # govulncheck
make smoke                # dry-run all scenarios
make public-smoke         # live public API smoke, requires BENCH_API_URL
make private-review-smoke # live private review write smoke
make ui-dev               # Vite dev server for local UI
make ui-build             # production UI build

See Testing Guide for local test coverage, CI gates, and smoke-test workflows.

License

Apache License 2.0

Ver o original no GitHub

Bom para

Relacionados