Trine Eval — Desenvolvimento Guiado por Avaliação com 3 Agentes
ats-kinoshita-iso/trine-eval
Orquestra 3 agentes para dev eval-driven com ciclo contrato→build→eval
Ver no GitHubArquitetura modular de três agentes (Planejador, Gerador, Avaliador) para desenvolvimento orientado por avaliação. Implementa o padrão Planner-Generator-Evaluator da Anthropic como plugin para Claude Code, com avaliação de sprints contra critérios de sucesso testáveis e comunicação via arquivo.
Instalação
npx convez-ai ats-kinoshita-iso/trine-eval1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Desenvolvimento orientado por testes com múltiplos agentes
- Decomposição automática de projetos em sprints testáveis
- Validação adversarial de implementações antes de produção
- Avaliação contínua contra rubricas customizáveis (web-app, API, RAG, CLI)
Veredito

Solução arquitetural robusta para equipes que querem automação inteligente de ciclos dev-test-eval. Exige Python e compreensão de multi-agent orchestration, mas oferece framework reutilizável e agnóstico a projeto com rubrics swappáveis.
README
A tradução para o português deste README ainda está na fila.
trine-eval
A modular three-agent eval-driven development harness for Claude Code. Implements Anthropic's Planner-Generator-Evaluator architecture as a portable, rubric-swappable plugin (plugins/trine-eval/), shipped alongside the trine_eval Python eval library (src/trine_eval/) that the harness builds and exercises.
How It Works
Three agents collaborate through files on disk — no shared context:
- Planner — Expands a short user prompt into a product spec with sprint decomposition
- Generator — Implements one sprint at a time, negotiating testable success criteria before writing code
- Evaluator — Adversarially tests each sprint against its contract, grading PASS/FAIL with specific evidence
The agents communicate exclusively through files in a .harness/ directory. Each sprint follows a contract→build→eval→retry cycle.
Installation
Copy or clone this repository into your Claude Code plugins directory, or install from a target project:
# From your project directory
claude /plugin install /path/to/trine-eval
Usage
1. Initialize the Harness
/harness-kickoff Build a task management API with user authentication and team workspaces
This will:
- Detect your project type (or ask you to choose)
- Create the
.harness/directory with configuration - Run the Planner to produce a spec and sprint plan
- Present the plan for your review
2. Run Sprints
/harness-sprint
Runs the next incomplete sprint through the full cycle:
- Generator proposes a sprint contract with testable success criteria
- Evaluator reviews the contract for testability and completeness
- Generator implements the sprint
- Evaluator tests against the contract
- If failures: Generator fixes, Evaluator re-tests (up to 3 rounds)
To target a specific sprint:
/harness-sprint 3
3. View Progress
/harness-summary
Generates a cross-sprint analysis with pass rates, trends, failure patterns, and recommendations.
Project Types and Rubrics
The harness ships with four rubrics. Set the project type during kickoff or in .harness/config.json:
| Type | Rubric | Key Dimensions |
|---|---|---|
web-app | web-app | Functionality, Visual Design, Code Quality, Robustness |
rag-system | rag-system | Retrieval Quality, Answer Faithfulness, System Robustness, Architecture |
cli-tool | cli-tool | Functionality, Usability, Error Handling, Code Quality |
api-service | api-service | Correctness, Robustness, API Design, Code Quality |
Configuration
.harness/config.json controls harness behavior:
{
"project_type": "web-app",
"rubric": "web-app",
"max_retries": 3,
"pass_threshold": {
"per_dimension_minimum": 2,
"critical_dimensions": ["functionality"],
"critical_minimum": 3
},
"contract_negotiation_rounds": 2,
"git_checkpoint": true,
"components_enabled": {
"planner": true,
"contract_negotiation": true,
"sprint_decomposition": true,
"eval_summary": true
}
}
Disabling Components
The components_enabled section lets you simplify the harness as models improve. For example, disabling contract_negotiation skips the Evaluator's contract review — the Generator's proposed criteria are accepted directly.
Phase 2 Configuration Knobs
These optional fields extend .harness/config.json with backward-compatible defaults. A config that omits them runs exactly as in Phase 1.
-
thinking.profile— one of"default","fast", or"thorough". Default:"default", which preserves Phase-1 behavior (no override applied to agent-level adaptive-thinking effort declared in agent frontmatter). The"fast"and"thorough"values are reserved for a future override dispatcher; the default is the only path that mutates today's behavior, and it does not. Each role's effort level is declared in the agent's own frontmatter (plugins/trine-eval/agents/planner.md,plugins/trine-eval/agents/generator.md,plugins/trine-eval/agents/evaluator.md, and theharness-summaryskill) —mediumfor routine planning and implementation,highfor capability evaluation, andmaxfor contract review and cross-sprint summary analysis. -
batch.enabled— boolean. Default:false. Whentrueand a sprint has at leastbatch.min_criteriacriteria, eval verifications are submitted as a single Anthropic Batch API call (50% discount on input/output tokens, 24-hour SLA). Batch is a cost optimization, not a latency optimization. With the defaultfalse, evaluations run synchronously as in Phase 1. -
batch.min_criteria— integer. Default:20. Sprints with fewer criteria stay synchronous even whenbatch.enabledistrue— the batch overhead is only worth absorbing on large suites. The criterion count compared against this threshold is the same count emitted insprint-{NN}.tasks.json(success criteria + Should-NOT gates). -
regression.enabled/regression.fail_fast— seerules/harness-conventions.md. The defaults are auto (truthy iffregression.jsonhas a non-emptytasksarray) andtrue(abort on regression failure). Pre-Phase-2 projects whose.harness/config.jsonlacks theregressionobject see no behavior change. -
trials/sandbox.mode/taxonomy.emit_tasks_json— see Sprint 6 documentation. Defaults1,"none",truereproduce Phase-1 behavior. -
evaluator_tools.playwright— string. Default"auto". Reserved values:"auto"(Playwright MCP enabled whenproject_type == "web-app", disabled otherwise — recommended default),"never"(disable unconditionally — used when running aweb-appproject without Playwright installed; the Evaluator falls back tocurland routes Visual Design findings to human review),"always"(enable regardless of project type — for explicit testing of the Playwright path). Backward compatibility: a config that lacks theevaluator_toolsobject hits"auto", which combined with any non-web-appproject type resolves to "Playwright disabled" — exactly Phase-1 behavior.
Adding Custom Rubrics
- Create a new file in
plugins/trine-eval/skills/eval-rubric/rubrics/your-type.md - Follow the structure of existing rubrics:
- Define quality dimensions with percentage weights (must total 100%)
- For each dimension, provide a 1-5 scoring table with specific descriptions
- Define hard thresholds (minimum scores that cause automatic failure)
- List testing tools appropriate for the domain
- Set
"rubric": "your-type"in.harness/config.json
Directory Structure
trine-eval/ # Marketplace root
├── .claude-plugin/marketplace.json # Marketplace manifest
├── plugins/trine-eval/ # The Claude Code plugin (methodology layer)
│ ├── .claude-plugin/plugin.json # Plugin manifest
│ ├── skills/
│ │ ├── harness-kickoff/ # Entry point: init + planning
│ │ ├── harness-sprint/ # Per-sprint contract→build→eval loop
│ │ ├── eval-rubric/ # Rubric loader + domain rubrics (incl. harness-build)
│ │ ├── sprint-contract/ # Contract template + negotiation protocol
│ │ ├── bootstrap-failures/ # Seed the eval suite from real failures
│ │ └── harness-summary/ # Cross-sprint analysis
│ ├── agents/
│ │ ├── planner.md # Product strategist
│ │ ├── generator.md # Senior engineer
│ │ └── evaluator.md # Adversarial QA
│ ├── hooks/hooks.json # Session lifecycle hooks
│ └── rules/harness-conventions.md # Conventions for .harness/ files
├── src/trine_eval/ # The Python eval library (runtime layer)
│ ├── runner/ models/ scorers/ # Async runner, Anthropic wrapper, scorers
│ └── observability/ # OpenTelemetry + Langfuse
├── tests/ # Library tests (uv run pytest)
└── pyproject.toml uv.lock # uv-managed Python project
When running in a target project:
your-project/
├── .harness/ # Created by harness-kickoff
│ ├── config.json # Project type, rubric, thresholds
│ ├── spec.md # Planner output
│ ├── sprints.json # Sprint decomposition
│ ├── contracts/sprint-NN.md # Negotiated sprint contracts
│ ├── evals/sprint-NN.md # Evaluation results per sprint
│ ├── progress.md # Session-resumable progress log
│ └── summary.md # Cross-sprint summary
Design Principles
- Files, not shared memory. Agents communicate via
.harness/files. This creates an audit trail and survives session interruptions. - Grade outcomes, not paths. The Evaluator checks what was produced, not how.
- Specific evidence required. Every FAIL must cite file paths, line numbers, and error messages.
- Complexity decreases over time. Components can be individually disabled as models improve.
- Rubrics are the parameterization point. Change the rubric to change the project type — orchestration stays the same.