claudemd-check
BenMalaga/claudemd-check
Testes de regressão para suas instruções CLAUDE.md com aderência por regra
Ver no GitHubAgent que executa testes de regressão para suas instruções CLAUDE.md, rodando sondagens em sessões Claude Code reais e medindo taxas de aderência por regra. Detecta quando edições ou atualizações de modelo quebraram silenciosamente suas regras de comportamento.
Instalação
npx convez-ai BenMalaga/claudemd-check1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Validar conformidade de agentes com instruções customizadas
- Detectar breaking changes em CLAUDE.md ou atualizações de modelo
- Medir e reportar taxas de aderência de regras antes/depois
- Integrar verificação de instruções em CI/CD
Veredito

Ferramenta especializada e madura para validação contínua de instruções de agentes Claude Code. Sem dependências npm, custo transparente, integração nativa em CI — essencial para equipes que mantêm CLAUDE.md em produção.
README
A tradução para o português deste README ainda está na fila.
claudemd-check
Regression tests for your CLAUDE.md.
Your instructions file says "always use pnpm", "never commit to main", "every new file ends with a license header". But do you actually know which rules your agent obeys, how often, and whether last week's edit (or a model update) silently broke rule 7?
Every Claude Code user has a CLAUDE.md. Almost nobody tests it. Rules accumulate, contradict, decay. The only feedback loop is vibes. There are eval harnesses for skills (skillgrade, promptfoo, Anthropic's skill-creator), but the instructions file itself (the thing every session loads) has no test runner.
claudemd-check is that runner. You write scenario probes (a prompt + the deterministic footprint an adherent run leaves behind), it executes each one N times against real headless Claude Code sessions in throwaway workspaces, and reports per-rule adherence rates, diffed against a saved baseline, wired for CI.
claudemd-check v0.1.0 · CLAUDE.md · 2 scenarios × 2 runs · model haiku
verified-trailer — New text files end with VERIFIED
✓ run 1 adhered
✓ run 2 adhered
adherence 100% (2/2, required 100%)
no-delicious — Never use the word delicious
✗ run 1: pie.txt contains forbidden /delicious/
✓ run 2 adhered
adherence 50% (1/2, required 100%)
vs baseline:
▼ no-delicious: 100% → 50%
1/2 scenarios met their adherence bar.
That ▼ 100% → 50% line is the product: the CLAUDE.md edit you made this
morning just lost you a rule, and you found out from a red CI check instead of
three weeks of subtle damage.
Install
npm install -g claudemd-check # or npx claudemd-check
Requires Claude Code (claude on your PATH)
and Node ≥ 18. Zero npm dependencies.
Cost honesty: every run is a real agent session billed to your Anthropic account/subscription. Write scenarios with
--runs 1 --model haiku, then raise runs for statistically meaningful rates. Small scenarios on haiku cost on the order of a cent each.
Writing scenarios
claudemd.test.json, next to the CLAUDE.md it tests
(full example):
{
"runsPerScenario": 3,
"model": "haiku",
"allowedTools": ["Write", "Edit", "Read", "Bash"],
"scenarios": [
{
"id": "always-pnpm",
"rule": "Always use pnpm, never npm",
"prompt": "add lodash to this project",
"files": { "package.json": "{\"name\":\"demo\"}" },
"assert": [
{ "type": "transcript_match", "pattern": "pnpm (add|install)" },
{ "type": "transcript_not_match", "pattern": "\\bnpm (install|i)\\b" },
{ "type": "file_absent", "path": "package-lock.json" }
]
}
]
}
Each run gets a fresh temp workspace (inline files and/or a fixture
directory, plus the CLAUDE.md under test), so runs never contaminate each other
(or your repo).
Assertions (all deterministic: no LLM judges, no flaky grading):
| Type | Checks |
|---|---|
transcript_match / transcript_not_match | Regex over the agent's own actions: its text and tool calls (commands run, files written). Deliberately excludes the rule text itself and tool results, so "never say X" isn't false-flagged by the agent reading the rule. |
file_exists / file_absent | Workspace state after the run. |
file_contains / file_not_contains | Regex over a produced file. |
command | Any shell command, run in the workspace; exit 0 = pass. The escape hatch that can verify anything. |
CI
- run: npx claudemd-check # exits 1 if any rule is below its bar
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
Save a baseline once (claudemd-check --save-baseline, commit it), and every
subsequent run prints per-rule deltas, so a CLAUDE.md edit or a new model
release that drops "never touch main" from 100% to 60% fails the build, with
the rule named.
Flags
--spec <file> · --claudemd <file> · --runs N · --model <name> ·
--save-baseline · --baseline <file> · --json
Scope
claudemd-check tests instruction adherence in Claude Code. It does not test skill triggering (use skillgrade or promptfoo for SKILL.md files) and does not grade output quality. It checks the deterministic footprint of rule-following, which is what you can actually regress on. Hooks and slash-command coverage are the roadmap.