convez.ai
AgenteAutomaçãoAvançado
dustinfox-code

Arbiter — Crítico Adversarial para Claude Code

dustinfox-code/arbiter

Loop iterativo com crítica adversarial integrada ao Claude Code

Ver no GitHub
InicialPluginScore · de 0 a 100

Agent que intercepta sinais de parada do Claude Code e valida se a tarefa foi realmente concluída antes de permitir o encerramento. Executa critérios customizáveis (testes, TypeScript, linting, revisão adversarial com IA) e reenvia Claude para correções com feedback estruturado até atingir os limites de iteração.

Instalação

$npx convez-ai dustinfox-code/arbiter

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
1
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 2 semanas
Versão
v0.1.0
Issues abertas
1
Contribuidores
1

Casos de uso

  • Validar automaticamente que testes passam antes de parar
  • Garantir zero erros TypeScript em tarefas de codificação
  • Executar revisão adversarial com personas de segurança e correção
  • Automatizar gate de qualidade com critérios customizados

Veredito

Ferramenta técnica sólida para quem usa Claude Code em workflows de desenvolvimento. Força rigor na conclusão de tarefas, mas exige setup cuidadoso de critérios e compreensão clara do ciclo de feedback para não ficar preso em loops infinitos.

README

A tradução para o português deste README ainda está na fila.

arbiter-cc

Arbiter intercepts Claude Code's stop signal and evaluates whether the task is actually done before letting the session end. If criteria aren't met, it sends Claude back to work with structured feedback about what's still failing.

Quick Start

npx arbiter-cc init                              # wire stop hook + create .arbiter/
npx arbiter-cc task start "add user login endpoint"  # register a task
# start Claude Code — Arbiter fires automatically when Claude tries to stop

That's it. init configures the Claude Code stop hook and creates .arbiter/config.json with defaults. task start registers a task with two default criteria: tests-pass and no-ts-errors. When Claude tries to stop, Arbiter evaluates the criteria and either allows the stop or sends Claude back.

To customize criteria before starting Claude, edit .arbiter/task.json:

npx arbiter-cc check    # dry-run: trigger the hook manually to test your setup

How the Loop Works

Claude tries to stop
    │
    ▼
Arbiter reads .arbiter/task.json
    │
    ├── no task?           → exit 0 (allow stop)
    ├── task complete/failed? → exit 0 (allow stop)
    ├── iteration >= max?  → mark failed, exit 0
    │
    ▼
Run all criteria (tests, tsc, custom commands, etc.)
    │
    ├── all required pass  → mark complete, exit 0
    │
    └── any required fail  → write feedback to stdout, exit 2 (loop)

Exit codes:

  • 0 — stop allowed (task complete, failed, or no task)
  • 2 — Claude re-enters with feedback injected as context
  • 1 — internal Arbiter error

Each iteration saves a checkpoint to .arbiter/checkpoints/ with the pass/fail state and git hash. The loop runs up to maxIterations times (default: 5), then marks the task failed and lets Claude stop.

Completion Criteria Types

Edit the criteria array in .arbiter/task.json to control what Arbiter checks. Each entry needs an id, type, description, required (boolean), and config object.

TypeWhat it doesConfig
tests-passRun test suite, pass if exit 0{ "command": "npm test" }
no-ts-errorsRun tsc --noEmit{ "tsconfigPath": "tsconfig.json" }
commandRun any shell command, pass if exit 0{ "command": "npm run lint" }
file-existsCheck a file exists (path-traversal safe){ "path": "src/auth.ts" }
llmCall Anthropic API with prompt + file context{ "prompt": "...", "files": ["src/auth.ts"], "model": "...", "timeout": 30000 }
criticRun adversarial review personas{ "personas": ["correctness", "security"], "files": "git-diff" }
customLoad a JS module as evaluator{ "path": "my-evaluator.mjs" }

Example: full criteria array

{
  "criteria": [
    {
      "id": "tests-pass",
      "type": "tests-pass",
      "description": "All tests pass",
      "required": true,
      "config": { "command": "npm test" }
    },
    {
      "id": "no-ts-errors",
      "type": "no-ts-errors",
      "description": "No TypeScript compilation errors",
      "required": true,
      "config": {}
    },
    {
      "id": "lint-clean",
      "type": "command",
      "description": "ESLint passes with no errors",
      "required": false,
      "config": { "command": "npx eslint src/ --max-warnings 0" }
    },
    {
      "id": "critic-review",
      "type": "critic",
      "description": "Adversarial review passes",
      "required": true,
      "config": { "personas": ["correctness", "security"], "files": "git-diff" }
    }
  ]
}

Set "required": false to make a criterion advisory-only (warnings reported but don't block completion).

llm criteria

The LLM evaluator sends your prompt plus file contents to the Anthropic API and expects a { "passed": boolean, "feedback": "..." } JSON response. Set "files": "git-diff" to automatically include changed files.

{
  "id": "docs-updated",
  "type": "llm",
  "description": "README reflects the new API",
  "required": false,
  "config": {
    "prompt": "Check if the README documents all exported functions. Return passed=false if any are missing.",
    "files": ["README.md", "src/index.ts"]
  }
}

Requires ANTHROPIC_API_KEY in the environment.

custom criteria

Export a default async function from a .mjs file:

// my-evaluator.mjs
export default async function(criteria, payload, cwd) {
  // do your checks
  return { passed: true, feedback: null }
  // or: { passed: false, feedback: "What went wrong" }
}
{
  "id": "my-check",
  "type": "custom",
  "description": "Custom project check",
  "required": true,
  "config": { "path": "my-evaluator.mjs" }
}

The Critic

The critic runs adversarial LLM-powered personas that review code for AI-specific failure modes. Each persona is a focused system prompt that receives your code and task description, returning structured findings.

Built-in personas

PersonaWhat it catches
correctnessSpec drift, incomplete features, TODOs, mismatched signatures
edge-casesMissing null checks, unhandled rejections, off-by-one, resource leaks
securityInjection, path traversal, hardcoded secrets, unsafe eval
hallucinationsImports that don't exist in package.json, fake API methods
test-qualityTautological tests, missing assertions, tests that can't fail

Run standalone

# Review git-diff files with default personas (correctness, edge-cases, security)
npx arbiter-cc review

# Pick specific personas
npx arbiter-cc review --personas correctness,hallucinations

# Review specific files
npx arbiter-cc review --files src/auth.ts,src/api.ts

# Review all project files
npx arbiter-cc review --all

The review command uses the active task description for context (if one exists). Exits non-zero if any errors are found.

Add as a loop criterion

Add a critic type entry to your task criteria:

{
  "id": "critic-review",
  "type": "critic",
  "description": "Adversarial review finds no errors",
  "required": true,
  "config": {
    "personas": ["correctness", "security", "hallucinations"],
    "files": "git-diff"
  }
}

If config.personas is omitted, it falls back to the personas listed in .arbiter/config.json.

Custom personas

Define custom personas in .arbiter/config.json:

{
  "critic": {
    "personas": ["correctness", "security", "api-contracts"],
    "customPersonas": [
      {
        "id": "api-contracts",
        "description": "Verify API request/response schemas match the spec",
        "systemPrompt": "You are reviewing code for API contract violations..."
      }
    ]
  }
}

Custom personas with the same id as a built-in persona override it.

Configuration Reference

.arbiter/config.json — all fields optional, shown with defaults:

{
  "maxIterations": 5,
  "checkpointDir": ".arbiter/checkpoints",
  "taskFile": ".arbiter/task.json",
  "llmModel": "claude-sonnet-4-20250514",
  "critic": {
    "enabled": false,
    "personas": ["correctness", "edge-cases", "security"],
    "failOnWarnings": false,
    "customPersonas": []
  }
}
FieldTypeDefaultDescription
maxIterationsnumber5Max loop iterations before task is marked failed
checkpointDirstring".arbiter/checkpoints"Where checkpoint JSON files are stored
taskFilestring".arbiter/task.json"Path to the active task file
llmModelstring"claude-sonnet-4-20250514"Model used for llm and critic evaluators
critic.enabledbooleanfalseWhether critic runs as part of the loop (when using critic criteria type, this is separate)
critic.personasstring[]["correctness", "edge-cases", "security"]Default personas for critic evaluations
critic.failOnWarningsbooleanfalseIf true, warnings (not just errors) cause critic to fail
critic.customPersonasarray[]Custom persona definitions (see above)

Missing config file = all defaults. Partial config = deep-merged with defaults.

CLI Reference

arbiter-cc init                          Initialize Arbiter in the current project
arbiter-cc task start "<description>"    Start a task (default criteria: tests-pass, no-ts-errors)
arbiter-cc task status                   Show current task state and last checkpoint
arbiter-cc task clear                    Remove the active task
arbiter-cc check                         Manually trigger stop hook evaluation
arbiter-cc checkpoints                   List all checkpoints
arbiter-cc checkpoints --restore <id>    Restore to a checkpoint via git checkout
arbiter-cc review                        Run standalone critic review

Flags:

  • --max-iterations <n> — Override max iterations on task start (default: 5)
  • --force — Replace an existing task without clearing first
  • --personas <list> — Comma-separated persona IDs for review
  • --files <list|git-diff|all> — Files to review (default: git-diff)
  • --all — Shorthand for --files all

Known Limitations (v0.1)

No task editing. You can't add or remove criteria from a running task via the CLI. Edit .arbiter/task.json directly. If you break the JSON, Arbiter will error on the next stop hook.

Test output parsing is heuristic. The tests-pass evaluator extracts failure context using regex pattern matching (looks for "FAIL", "Error:", "AssertionError" near failure lines). Non-standard test output formats may produce poor feedback.

File size limits. Files over 1 MB are skipped. Critic context is capped at ~500 lines per file. Large files may be truncated or excluded from review.

Git-diff default can review nothing. Both the critic and files: "git-diff" resolve changed files via git diff --name-only HEAD. If there are no uncommitted changes (e.g., everything was committed), the file list is empty and the review is a no-op.

LLM/critic require ANTHROPIC_API_KEY. Not checked at init time — you'll get a failure at evaluation time if it's missing.

Custom evaluators are unsandboxed. The custom evaluator type runs your JS module with full filesystem and network access. The only guardrail is a 30-second timeout.

No atomic task state. Task status, checkpoint writes, and task updates are separate file operations. A process kill mid-loop could leave .arbiter/task.json in an inconsistent state. Run arbiter-cc task clear and start over if this happens.

Iteration limit is a hard stop. When maxIterations is reached, the task is marked failed and cannot be resumed. You must task clear and task start again.

Single dependency. Requires @anthropic-ai/sdk (^0.52.0). Node >= 18.

License

MIT

Ver o original no GitHub

Bom para

Relacionados