convez.ai
SkillProdutividadeIntermediário
pedroreisper

did-it-actually

pedroreisper/did-it-actually

Audita o trabalho do Claude com critérios formais e força correção automática

Ver no GitHub
InicialPluginScore · de 0 a 100

Skill que verifica automaticamente o próprio trabalho do Claude contra a solicitação do usuário antes de relatar conclusão. Fecha o loop com um crítico de contexto fresco, contrato de aceitação e loop de correção iterativo (máx. 3 iterações).

Instalação

$npx convez-ai pedroreisper/did-it-actually

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 4 meses
Versão
v1.0.0
Issues abertas
0
Contribuidores
1

Casos de uso

  • Validar se tarefas de código foram realmente completadas
  • Garantir que todos os critérios de aceitação foram atendidos
  • Evitar loops repetitivos de correção manual
  • Auditar PRs, testes e arquivos gerados antes de marcar como done

Veredito

Skill sofisticado que resolve um problema real: o loop de feedback deficiente entre humano e IA. O mecanismo de contrato formal + crítico independente + cap mecânico de 3 iterações é bem pensado e reduz drasticamente retrabalho em tarefas de desenvolvimento. Recomendado para quem trabalha com Claude em projetos que exigem verificação rigorosa.

README

A tradução para o português deste README ainda está na fila.

did-it-actually

Stop telling Claude the same thing five times. This skill makes Claude check its own work against your request — and fix the gap before reporting done.

License: MIT

The pain it solves

You ask Claude for a thing. Claude reports done. You read the result, find the gap, repeat the ask. Claude reports done again. You read the result, find a different gap. Three rounds later you give up and fix it yourself.

This is not a Claude problem — it's a feedback-loop problem. The implementer cannot reliably audit itself in the same context that produced the work. Self-review converges on prior belief.

did-it-actually closes the loop with three structural fixes:

  1. A formal contract — your request decomposed into discrete, falsifiable acceptance criteria written to .did-it-actually/contract.yml.
  2. A fresh-context critic — an independent sub-agent that audits the work without seeing the conversation that produced it.
  3. A fix-and-re-audit loop — the skill tells Claude to fix any NOT VERIFIED gaps and re-run the audit; the script enforces a hard cap of 3 iterations. The loop itself is prompt-driven (Claude reads the report and acts), but the cap, the verdict, and the contract integrity are mechanical.

The verdict is a pure function of evidence. Every ✅ cites a SHA, an exit code, or a file:line. Anything else is mechanically downgraded to ❌.

Quick demo

You: Add rate limit to /api/login, 5 req/min/IP. Add a test. Open a PR.

Claude: [edits 3 files, runs tests, says "done — PR opened"]

→ Stop hook fires → did-it-actually runs automatically

SELF-AUDIT REPORT
─────────────────────────────────────────────
Iteration: 1/3
✅ VERIFIED
  • rate-limit-module-exists  bytes:412 sha:a8c9...
  • login-imports-rate-limit  read:src/api/login.ts:3-3 matches /from ".\/rate-limit"/
  • tests-pass                exit:0 cmd:"npm test"
❌ BROKEN CLAIMS
  • pr-opened                 no gh pr create in claim ledger; gh pr view exits 1
VERDICT: NOT VERIFIED
        sub-ask dropped: PR was claimed but not opened.

→ Claude reads report → runs `gh pr create` → re-audits

Iteration: 2/3
VERDICT: VERIFIED

Claude: Done. PR opened: https://github.com/.../pull/47. All 43 tests pass.

You typed the request once. The audit caught the dropped sub-ask. Claude fixed it without being told. You got the result you wanted on the first reply.

Install

Inspect-then-run (recommended — install.sh is short and worth a read):

git clone https://github.com/pedroreisper/did-it-actually
cd did-it-actually
less install.sh       # 80 lines, no obfuscation
bash install.sh --hook

One-liner (if you trust the source):

curl -fsSL https://raw.githubusercontent.com/pedroreisper/did-it-actually/main/install.sh | bash -s -- --hook

Project-scoped (shared with a team via your repo):

bash install.sh --project --hook

Verify it's wired up:

bash ~/.claude/skills/did-it-actually/scripts/doctor.sh

The --hook flag enables proactive firing (recommended). Without it, the skill only runs when you invoke it explicitly.

Usage

Manual

Type any of these in chat:

  • /did-it-actually
  • "did it actually finish?"
  • "audita-te"
  • "verifica o que fizeste"
  • "is it really done?"

Or pass the canonical request explicitly:

/did-it-actually "the thing I asked you to do"

Proactive (recommended)

With --hook installed, the audit fires automatically before Claude emits any "done / completo / ready / shipped" message on a turn that touched ≥2 files or used ≥3 tool calls. You don't have to remember.

Per-PR

Audit a whole branch against the PR description as the contract:

bash ~/.claude/skills/did-it-actually/scripts/audit.sh run --against=main

What this skill is NOT

  • Not a code reviewer (/code-review does that)
  • Not a security scanner (/security-review does that)
  • Not a test framework (uses your existing test runner)
  • Not an auto-fixer for code-quality nits — it audits request fidelity, not style

This skill is intentionally narrow. It answers exactly one question: did Claude actually do what you asked.

How it works

┌─────────────────────────────────────────────────────────────┐
│  User request                                                │
│       │                                                      │
│       ▼                                                      │
│  contract.yml ◄────────────────────────────────┐            │
│       │              (Step 1: decompose ask)    │            │
│       ▼                                          │            │
│  Claude does the work, appending claims.jsonl   │            │
│       │              (Steps 2-3)                 │            │
│       ▼                                          │            │
│  audit.sh run                                    │            │
│   ├─► deterministic checks                       │            │
│   │     - ledger ≡ git diff                      │            │
│   │     - existence / content / command          │            │
│   │     - syntactic-rot scan                     │            │
│   │                                              │            │
│   └─► Task spawn → fresh-context critic ────────┘            │
│                       (no conversation history)              │
│                                                              │
│  report.json + prose render                                  │
│       │                                                      │
│       ├── VERIFIED ───────────► report done to user          │
│       └── NOT VERIFIED ───────► loop: fix → re-audit         │
│                                  (cap: 3 iterations)         │
└─────────────────────────────────────────────────────────────┘

Full architecture in SKILL.md.

Anti-gaming guarantees

The skill is structurally hostile to the failure modes that make naive self-audits theater:

Failure modeHow the skill blocks it
Same-context judgingCritic runs in a fresh sub-agent with no conversation history
Phantom ✅ ("I checked and it's fine")Every ✅ requires citable evidence (SHA, exit code, file:line)
Moving the goalposts mid-iterationContract SHA is recorded; mutation → AUDIT_FAILED
Infinite "fix → re-audit" loopsHard 3-iteration cap enforced by script, not by prompt
Skipping the auditStop hook fires regardless of agent intent; doctor proves it's wired
Stub bodies passing as workAST walk catches pass/unimplemented!()/TODO() patterns beyond regex
Silenced testsDiff vs pre-state catches new skips, not pre-existing ones
Swallowed Bash failuresscan_session.py reads the session JSONL for unaddressed non-zero exits

Validation

The skill ships with an eval harness (tests/) measuring recall, precision, and calibration against 9 synthetic failure-mode fixtures. Ship gates:

  • recall(missing-file) ≥ 0.95
  • recall(dropped-subask) ≥ 0.95
  • recall(stub-body) ≥ 0.85
  • precision(NOT VERIFIED) ≥ 0.90
  • calibration(VERIFIED) ≥ 0.95
  • cost ≤ 5,000 tokens / 15 tool calls per audit
bash tests/run-evals.sh

See references/eval-harness.md for methodology.

Layout

did-it-actually/
├── SKILL.md                    # entry point Claude Code loads
├── README.md                   # this file
├── LICENSE                     # MIT
├── install.sh                  # one-liner installer
├── references/
│   ├── contract-format.md      # contract.yml YAML schema
│   ├── checks.json             # machine-readable check definitions
│   ├── output-schema.json      # JSON Schema for report.json
│   ├── audit-checklist.md      # per-check intent + risks
│   ├── edge-cases.md           # no-git, monorepos, generated files, CI-only
│   └── eval-harness.md         # how to measure the skill
├── scripts/
│   ├── audit.sh                # main driver (init / run / render / status / doctor)
│   ├── render.py               # JSON → prose verdict box
│   ├── snapshot.sh             # pre/post state capture
│   ├── snapshot.py             # snapshot diff utilities
│   ├── claim.sh                # append a single claim row
│   ├── scan_session.py         # reconstruct ledger from session JSONL
│   ├── critic.md               # prompt template for the fresh-context critic
│   └── doctor.sh               # self-diagnostic
├── hooks/
│   ├── stop_self_audit.py      # Stop hook for proactive firing
│   └── install_hook.sh         # wires the hook into ~/.claude/settings.json
├── examples/
│   ├── clean-run.md            # VERIFIED scenario
│   ├── needs-fixes.md          # NOT VERIFIED → followups
│   ├── auto-iteration.md       # the killer "no more 10 times" flow
│   └── contract.example.yml    # sample contract
└── tests/
    ├── run-evals.sh            # eval harness driver
    └── fixtures/               # synthetic failure-mode test cases

Contributing

Two things especially valuable:

  1. New fixtures — when a real Claude session produces a failure mode this skill doesn't catch, distill it into a fixture under tests/fixtures/ and open a PR. The eval harness grows with the failure surface.
  2. AST checks for new languages — references/checks.json lists per-language stub-body AST checks. Adding Rust, Go, Kotlin, Swift, Ruby AST walks reduces false-negative rate on those ecosystems.

Keep SKILL.md under ~200 lines. Detail goes in references/.

License

MIT.

Acknowledgments

The architecture distills critiques from six independent reviewer perspectives — LLM-eval research, SRE/observability, DX/OSS adoption, adversarial-security, Anthropic-skill ecosystem, and formal-verification — applied to an earlier prose-only draft. The convergent insight: structure over narration, evidence over claims, fresh context over self-review, auto-iterate over notify.

If this skill saves you one "please can you actually fix it this time" reply, it's worth its install cost.

Ver o original no GitHub

Bom para

Relacionados