convez.ai
MCP ServerDevOps & InfraestruturaAvançado
tomas-samek

LLM Framework Benchmark

tomas-samek/llm-framework-benchmark

Benchmark oracle-graded para testar agentes IA em frameworks desconhecidos

Ver no GitHub
InicialPluginScore · de 0 a 100

MCP server que funciona como bancada de testes para medir a capacidade de agentes de IA construírem a mesma especificação em diferentes stacks. Usa um oráculo externo (independente) para avaliar conformidade, eficiência e qualidade de código. Caso de estudo: Spring Boot vs Tiko DI em serviço event-driven Kafka → H2.

Instalação

$npx convez-ai tomas-samek/llm-framework-benchmark

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
1
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 1 semana
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Avaliar performance de agentes de IA em frameworks fora do corpus de treinamento
  • Medir impacto de documentação in-repo (e.g., CLAUDE.md) vs familiarity da modelo
  • Detectar regressões silenciosas entre versões maiores (ex: Spring Boot 3.x vs 4.x)
  • Comparar efetividade de affordances (MCP, guidance files) em scaffolds framework-neutral

Veredito

Benchmark rigoroso e insightful que revela gaps críticos na performance de agentes IA — especialmente o fenômeno de falha silenciosa em versões maiores desconhecidas. Prova empiricamente que documentação in-repo estruturada bate training-familiarity; recurso valioso para arquitetos que precisam de evidence-based guidance em escolhas de stack.

README

A tradução para o português deste README ainda está na fila.

LLM Framework Benchmark

How well can an AI coding agent build the same specified system on different frameworks? A framework-neutral spec, an external black-box oracle that grades the result, and clean per-framework scaffolds — so the only variable is the framework (and the agent). First comparison: Spring Boot vs Tiko DI, on an event-driven Kafka → H2 → merged-notification service.

The benchmark grades each run on compliance (does it actually pass the acceptance scenarios, judged by an independent harness — not the agent's own tests), with hooks for efficiency and code quality.

Note on Tiko — intentionally unknown to LLMs. Tiko DI is purposefully developed and used as an out-of-training-corpus framework. Its premise is that a framework can be AI-friendly without relying on the model's prior familiarity — instead shipping machine-readable, in-repo affordances: guidance (CLAUDE.md, .ai-skills/), compile-time wiring validation, and an MCP topology server. So in this benchmark Tiko is a deliberate test case for the question "can an agent succeed on a framework it was never trained on, given good in-repo affordances?" — which is also why it makes a fair foil for a brand-new, equally-unfamiliar Spring Boot 4.0.6. (Stage-1 answer: largely yes — 86% median, vs Boot-4's 0%.)

Headline results — Stage 1

Model: Claude Sonnet 4.6 contestants, orchestrated by Claude Opus 4.8 (2026-06; corrected — an earlier revision mis-attributed contestants to Opus 4.8; see the model column in results/metrics.csv). N=5 per cell (spring3 is a single reference run). Compliance = oracle scenarios passed / 7. Fixed fixtures, fresh broker per trial. Full write-up: results/RESULTS.md.

CellFramework / versionMedian complianceNotes
springSpring Boot 4.0.60% (1/5 correct)brand-new major; model writes Boot-3 idioms that silently no-op
spring3Spring Boot 3.3.5100% (n=1, first try)a version the model knows → nails it
tikoTiko 0.2.286%out-of-corpus framework; carried by bundled in-repo guidance
tiko-mcpTiko 0.2.2 + MCP86%identical to tiko here (see finding 3)

What it found

  1. Version recency dominates. The same model scored 100% on Spring Boot 3.3.5 and a median 0% on 4.0.6. A single major-version bump — not a worse model, not a worse framework — caused it. The dangerous part is silent failure: Boot-4 apps compiled, ran, and emitted nothing. A current-generation model shows this; it's structural to training-on-historical-corpus, and the lag is longer than "time since cutoff" because the old version dominates the corpus.

  2. An unknown framework that ships its own docs can beat a known framework's unknown new version. Tiko (essentially absent from training) reached 86% median because its scaffold ships machine-readable guidance the agent reads at build time; bare Boot-4 shipped neither training familiarity nor docs.

  3. The MCP topology server showed no compliance lift here — tiko and tiko-mcp were identical (0 0 86 86 100). The pilot's apparent +57 pts was a fixture-timing artifact. Caveat: the validation gate targeted wiring, which the model already got right; it did not target config, which is where Tiko actually failed — so this is "MCP-for-wiring = no lift," not a general verdict.

  4. Tiko's main AI-friendliness issue is strict config validation — 4/10 Tiko trials died at startup because the model used kebab-case / Spring-style config keys that Tiko's @Configuration records reject. The single most actionable fix.

These are directional results: small N, one model, one point in time, and some early prompts carried hints. See results/RESULTS.md → "Caveats / validity".

How it works

  • Framework-neutral spec (docs/specs/stage-1-spec.md) — describes what to build, with zero framework words.
  • External conformance oracle (conformance/oracle/, Java) — brings up Kafka, publishes the shared fixtures (fixtures/stage-1/scenarios.json), drains the output topic, and asserts the result. The agent's own tests don't count.
  • Asymmetric-native fairness — each framework uses its own first-party stack; neither may copy from example repos. Tiko, having no first-party DB/HTTP module, must hand-build those. Full rules in docs/benchmark-protocol.md.
  • Pinned substrate — apache/kafka 3.8.0, H2 2.2.224, Lucene 9.11.1, JDK 21, identical for everyone, so the only variable is the framework + the agent.

Run it — including with other agents

See RUNNING.md for the canonical, neutral per-trial procedure: copy a golden scaffold, hand the agent only the spec + dependency rules (no hints), let it build, then grade with the oracle. The procedure is agent-agnostic — point Claude, GPT, Gemini, Cursor, etc. at the same scaffold + spec and compare. Running it across agents is the natural next step.

Layout

docs/   benchmark-protocol.md (rules) + specs/ (framework-neutral WHAT)
fixtures/   shared acceptance vectors (input -> expected output)
conformance/   oracle (Java) + docker-compose Kafka + run scripts
scaffolds/   golden per-framework starting points (+ GENERATE.md)
results/   metrics.csv + per-trial oracle JSON + RESULTS.md
runs/   per-trial workspaces (git-ignored; see runs/README.md)

Status

  • Protocol, Stage-1 spec, fixtures, conformance oracle (tests green)
  • Golden scaffolds (Spring Boot 4.0.6 + 3.3.5 reference; Tiko 0.2.2)
  • Stage-1 run: spring, tiko, tiko-mcp (5x) + spring3 reference — results/RESULTS.md
  • Run with non-Claude agents
  • Capture efficiency + code-quality rubric; n=10
  • Stage-2 spec (full-text search)

License

MIT — see LICENSE.

Ver o original no GitHub

Bom para

Relacionados