convez.ai
MCP ServerDevOps & InfraestruturaIntermediário
wjlgatech

CLI-Judge

wjlgatech/cli-judge

Teste real para ferramentas de IA: verifica o que funciona, não só como parece

Ver no GitHub
InicialPluginScore · de 0 a 100

Benchmark e harness de verificação para CLIs nativas de agentes, servidores MCP e harnesses de software. CLI-Judge testa se ferramentas de IA realmente funcionam em situações reais, não apenas se parecem corretas — avaliando 945+ cenários extraídos de relatórios de bugs autênticos.

Instalação

$npx convez-ai wjlgatech/cli-judge

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
1
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Validar CLIs geradas por agentes antes de produção
  • Testar MCP servers quanto a correção e segurança
  • Auditar ferramentas de IA em cenários reais gravados
  • Detectar falhas ocultas e comportamentos inseguros em software

Veredito

Ferramenta crítica para validação de agentes e ferramentas CLI em produção. Preenche uma lacuna real testando comportamento real versus aparência de código, com base em 945+ casos reais de falha.

README

A tradução para o português deste README ainda está na fila.


🤔 Why CLI-Judge exists (explain it to a 15-year-old)

AI agents are starting to do real things for us — book stuff, edit files, call services. They don't click buttons like we do; they run little command-line programs called tools.

Two popular "factories" mass-produce these tools for agents: CLI Printing Press and CLI-Anything. They're great at making tools fast.

But here's the catch: each factory grades its own homework — and it grades the wrong thing. It checks whether the tool looks right (does the code have the right shapes and patterns?) instead of whether the tool actually works (does it give the correct answer, safely, when it meets the real world?).

It's like a driving school that hands you a license for correctly naming the parts of a car — without ever checking whether you can drive down a real road without crashing. 🚗💥

So the tools pass their own tests… then break in real life: you ask for page 2 and get page 1, the tool mistakes a password for the wrong kind of key, it spits out raw gibberish instead of clean data, it crashes on Windows — or it quietly runs a hidden delete command hidden inside a filename.

CLI-Judge is the independent road test. It doesn't build tools, and it doesn't trust their looks. It replays real recorded situations at each tool and grades the result: Did you get the right answer? Did you stay safe? Were you fast and cheap? Every single point is backed by evidence from a real interaction — distilled from 945+ real bug reports these factories' own users actually filed.

That's the whole pitch you can give a stranger in 20 seconds:

"AI agents run tools. The people who make those tools grade their own homework on how the code looks — not whether it works. CLI-Judge is the referee that replays real situations and grades the real result, especially safety."


💡 The 5 reasons, in one breath

  1. An agent is only as smart as its tools. A broken tool quietly hands back wrong answers — and you trust them.
  2. Today's report cards grade looks, not results. A scorecard that checks "is the pattern in the source?" can't catch the bug a real user just hit.
  3. You can't be the player and the referee. A tool's own factory will never fail it the way an independent judge will.
  4. Real situations, real outcomes. Every score traces to a real recorded request or a real run — never a guess, never "the code looks fine."
  5. Safety is non-negotiable. One hidden destructive command can wreck a system, so an unsafe tool can never score above a C — no matter how correct or pretty it is.

🖼️ The problem in one picture

Structure scorecards (today)CLI-Judge (outcomes)
Question it asks"Does the code look right?""Does the tool work right?"
Evidence for a pointa pattern exists in the sourcea real replayed request / real run
Catches the bug a user actually hit?❌ usually not✅ that's the entire point
Who's grading?the tool's own factoryan independent judge
Pass it by editing a comment?yes 😬never

🎯 How it scores: 5 dimensions, 100 points

Every tool is replayed against real fixtures and scored on five dimensions (full point tables in RUBRIC.md):

DimensionWhat it measuresPtsA real failure it catches
D1 · Correctnessright typed output against a real captured payload35you ask for page 2, the tool sends a cursor instead of skip=50
D2 · Non-interactiveworks piped, --json, no prompts, typed exit codes20the tool blocks forever waiting on a human who isn't there
D3 · Cross-platformpaths / locale / encoding, and loud failure when a backend is missing15a Windows path breaks and it silently emits wrong output
D4 · Safety 🛡️declares blast radius, gates destructive ops, signs receipts, blocks injection20a malicious filename smuggles in a hidden os.system('…')
D5 · Efficiencytokens-per-task and latency vs a budget10--compact claims 70% smaller… but isn't

🛡️ Hard safety gate: any safety blocker caps the final grade at C — an unsafe tool is never shippable, no matter how correct it is everywhere else.


🚀 Quick Start

cd harness
python3 -m venv .venv && source .venv/bin/activate
pip install -e .                     # one hard dependency: jsonschema

cli-judge validate ../fixtures       # check every fixture against its schema
cli-judge run --adapter examples/echo_adapter.py --suite full   # score a tool → report.json + scorecard.md + a letter grade

You get a human scorecard.md, a machine report.json, and a one-line verdict like Grade: B (78.0/100) — deterministic, offline, and traceable to real evidence. Point CLI-Judge at a real tool by writing a tiny adapter (one invoke(call) -> Result function); ready-made adapters for the two factories live in harness/adapters/.


🧪 What it catches (from real bug reports)

Every fixture is distilled from a real, observed failure in one of the two factories' public issue trackers — so the benchmark is auditable, not invented:

FailureDimensionFrom a real issue
Pagination param drift (cursor vs skip/limit)D1cli-printing-press#2961
Short token misclassified as a JWTD1pp#1598
--compact leaks raw map[...] stringsD1pp#2950
Non-interactive mode broken (blocks on a prompt)D2CLI-Anything#214
Headless LibreOffice crashes on macOSD3ca#221
Code injection via an unsanitized macro pathD4ca#282
Token-file path traversal / symlink escapeD4ca#304

See the full list in fixtures/CATALOG.md.


🧭 The Reality Principle (the one rule)

Every point CLI-Judge awards must trace to (a) the tool's real stdout/stderr/exit-code from an actual run, (b) its behavior replayed against a recorded real payload, or (c) an independently verifiable property of its declared capability envelope.

A point is never awarded because "a pattern exists in the source code." That is exactly the trap CLI-Judge exists to escape.

The core suite runs fully offline against recorded fixtures (no network). Destructive-action tests are record/assert only — CLI-Judge never executes a dangerous command against a live system. Safe by construction.


📦 What's in the box

PathPurpose
SPEC.mdThe benchmark spec — dimensions, task format, fixture format
RUBRIC.mdThe 100-point rubric with exact point allocations and pass gates
harness/The runnable reference harness — replay engine, 5 scorers, signed receipts, runner, CLI
fixtures/Real-world failure fixtures distilled from the two issue trackers
schemas/JSON Schemas for tasks, fixtures, capability envelope, signed receipts
AGENTS.mdBuild brief for coding agents — architecture, work breakdown, definition of done
docs/Strategy, threat model, methodology, roadmap, FAQ

🚫 Scope discipline (read this)

CLI-Judge measures tools. It is not another generator and must never become one. If a contribution adds a code generator, it's out of scope. The defensible position — the whole reason this exists — is being the referee, not a third player on the field.


📜 License

Apache-2.0. Built to be cited, forked, and run against anyone's tools — including ours.

Ver o original no GitHub

Bom para

Relacionados