convez.ai
AgenteDevOps & InfraestruturaAvançado
BenMalaga

claudemd-check

BenMalaga/claudemd-check

Testes de regressão para suas instruções CLAUDE.md com aderência por regra

Ver no GitHub
InicialPluginScore · de 0 a 100

Agent que executa testes de regressão para suas instruções CLAUDE.md, rodando sondagens em sessões Claude Code reais e medindo taxas de aderência por regra. Detecta quando edições ou atualizações de modelo quebraram silenciosamente suas regras de comportamento.

Instalação

$npx convez-ai BenMalaga/claudemd-check

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 4 meses
Versão
v0.1.0
Issues abertas
0
Contribuidores
1

Casos de uso

  • Validar conformidade de agentes com instruções customizadas
  • Detectar breaking changes em CLAUDE.md ou atualizações de modelo
  • Medir e reportar taxas de aderência de regras antes/depois
  • Integrar verificação de instruções em CI/CD

Veredito

Ferramenta especializada e madura para validação contínua de instruções de agentes Claude Code. Sem dependências npm, custo transparente, integração nativa em CI — essencial para equipes que mantêm CLAUDE.md em produção.

README

A tradução para o português deste README ainda está na fila.

claudemd-check

Regression tests for your CLAUDE.md.

Your instructions file says "always use pnpm", "never commit to main", "every new file ends with a license header". But do you actually know which rules your agent obeys, how often, and whether last week's edit (or a model update) silently broke rule 7?

npm CI node deps license


Every Claude Code user has a CLAUDE.md. Almost nobody tests it. Rules accumulate, contradict, decay. The only feedback loop is vibes. There are eval harnesses for skills (skillgrade, promptfoo, Anthropic's skill-creator), but the instructions file itself (the thing every session loads) has no test runner.

claudemd-check is that runner. You write scenario probes (a prompt + the deterministic footprint an adherent run leaves behind), it executes each one N times against real headless Claude Code sessions in throwaway workspaces, and reports per-rule adherence rates, diffed against a saved baseline, wired for CI.

claudemd-check v0.1.0 · CLAUDE.md · 2 scenarios × 2 runs · model haiku

verified-trailer — New text files end with VERIFIED
  ✓ run 1 adhered
  ✓ run 2 adhered
  adherence 100% (2/2, required 100%)

no-delicious — Never use the word delicious
  ✗ run 1: pie.txt contains forbidden /delicious/
  ✓ run 2 adhered
  adherence 50% (1/2, required 100%)

vs baseline:
  ▼ no-delicious: 100% → 50%

1/2 scenarios met their adherence bar.

That ▼ 100% → 50% line is the product: the CLAUDE.md edit you made this morning just lost you a rule, and you found out from a red CI check instead of three weeks of subtle damage.

Install

npm install -g claudemd-check     # or npx claudemd-check

Requires Claude Code (claude on your PATH) and Node ≥ 18. Zero npm dependencies.

Cost honesty: every run is a real agent session billed to your Anthropic account/subscription. Write scenarios with --runs 1 --model haiku, then raise runs for statistically meaningful rates. Small scenarios on haiku cost on the order of a cent each.

Writing scenarios

claudemd.test.json, next to the CLAUDE.md it tests (full example):

{
  "runsPerScenario": 3,
  "model": "haiku",
  "allowedTools": ["Write", "Edit", "Read", "Bash"],
  "scenarios": [
    {
      "id": "always-pnpm",
      "rule": "Always use pnpm, never npm",
      "prompt": "add lodash to this project",
      "files": { "package.json": "{\"name\":\"demo\"}" },
      "assert": [
        { "type": "transcript_match", "pattern": "pnpm (add|install)" },
        { "type": "transcript_not_match", "pattern": "\\bnpm (install|i)\\b" },
        { "type": "file_absent", "path": "package-lock.json" }
      ]
    }
  ]
}

Each run gets a fresh temp workspace (inline files and/or a fixture directory, plus the CLAUDE.md under test), so runs never contaminate each other (or your repo).

Assertions (all deterministic: no LLM judges, no flaky grading):

TypeChecks
transcript_match / transcript_not_matchRegex over the agent's own actions: its text and tool calls (commands run, files written). Deliberately excludes the rule text itself and tool results, so "never say X" isn't false-flagged by the agent reading the rule.
file_exists / file_absentWorkspace state after the run.
file_contains / file_not_containsRegex over a produced file.
commandAny shell command, run in the workspace; exit 0 = pass. The escape hatch that can verify anything.

CI

- run: npx claudemd-check            # exits 1 if any rule is below its bar
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Save a baseline once (claudemd-check --save-baseline, commit it), and every subsequent run prints per-rule deltas, so a CLAUDE.md edit or a new model release that drops "never touch main" from 100% to 60% fails the build, with the rule named.

Flags

--spec <file> · --claudemd <file> · --runs N · --model <name> · --save-baseline · --baseline <file> · --json

Scope

claudemd-check tests instruction adherence in Claude Code. It does not test skill triggering (use skillgrade or promptfoo for SKILL.md files) and does not grade output quality. It checks the deterministic footprint of rule-following, which is what you can actually regress on. Hooks and slash-command coverage are the roadmap.

License

MIT

Ver o original no GitHub

Bom para

Relacionados