convez.ai
MCP ServerDevOps & InfraestruturaAvançado
jeremylongshore

J-Rig Binary Eval

jeremylongshore/j-rig-skill-binary-eval

Avaliação binária e rigorosa para skills do Claude antes de produção

Ver no GitHub
BomPluginScore · de 0 a 100

Harness de avaliação em sete camadas para skills, plugins e agents do Claude que aplica disciplina de release em nível de produção. Avalia integridade de pacote, qualidade de disparo, funcionalidade, proteção de regressão, valor baseline, variância de modelo e segurança de rollout através de critérios binários (sim/não) com avaliadores externos independentes.

Instalação

$npx convez-ai jeremylongshore/j-rig-skill-binary-eval

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
3
Forks
0
Acompanhando
0
Saúde
100%
Último commit
há 2 dias
Versão
v2.1.0
Issues abertas
21
Contribuidores
3

Casos de uso

  • Validar skills antes de merge em repositórios compartilhados
  • Detectar regressões causadas por atualizações de modelo
  • Medir precisão de triggers e evitar acionamentos acidentais
  • Gerar evidências auditáveis para decisões de rollout

Veredito

Ferramenta sólida para quem precisa de garantias de qualidade em skills Claude em produção. Exige investimento em infraestrutura de testes e CI/CD, mas elimina a improviso e garante que mudanças não quebram silenciosamente.

README

A tradução para o português deste README ainda está na fila.

j-rig-binary-eval

Part of the Intent Eval Platform — the umbrella mapping the six repos that converge via a shared Evidence Bundle schema.

Software-grade release discipline for Claude Skills

Binary evaluation harness that treats SKILL.md artifacts as production software. Package integrity, trigger precision, functional quality, regression gating, baseline comparison, model-aware testing, and evidence-backed rollout decisions — all through binary yes/no criteria with external evaluators.

License CI Release

Links: Master Blueprint · Epic Index · Doc Index · Contributing · Security


One-Pager

The Problem

Claude Skills ship on instinct. A skill author writes a SKILL.md, eyeballs it, maybe runs it once, and pushes. There is no regression gate, no trigger precision measurement, no baseline comparison, no model-variance tracking, and no evidence trail for rollout decisions. When a skill breaks silently after a model update or a description tweak causes sibling confusion across a pack, nobody knows until users complain.

The Solution

J-Rig Binary Eval is a seven-layer evaluation harness that scores every skill change across seven product surfaces before it ships:

  1. Package Integrity — Does it parse, validate, and reference real files?
  2. Trigger Quality — Does it fire on the right prompts and stay silent on the wrong ones?
  3. Functional Quality — Does it complete its task and produce correct artifacts?
  4. Regression Protection — Did this change break anything that previously worked?
  5. Baseline Value — Does the skill actually outperform the naked model?
  6. Model Variance — Does it work across Haiku, Sonnet, and Opus?
  7. Rollout Safety — Any prompt leakage, overreach, or unsafe automation?

Every criterion is binary (yes/no). The evaluator is always separate from the skill under test. Observed behavior outranks claimed behavior.

W5

WhoClaude Skill authors, skill pack maintainers, enterprise skill library operators
WhatEvaluation harness + regression gate + optimization engine for Claude Skills
WhereLocal CLI (author workflow), CI/CD (PR gate), team dashboard (reporting)
WhenEvery skill change: new skill, description edit, body rewrite, model update
WhySkills are production software — they need release-quality discipline, not vibes

Stack

LayerTechnology
RuntimeTypeScript, Node.js 20+, pnpm
CLI/Parsingcommander, @clack/prompts, picocolors, yaml, unified/remark
Validationzod
LLM Integration@anthropic-ai/sdk
Persistencebetter-sqlite3, drizzle-orm
Concurrencyp-limit, async-retry
Artifact Extractionpdf-parse, mammoth
Dashboard (future)Next.js, Tailwind, shadcn/ui

Key Differentiators

  • Binary criteria only — if a criterion can't be answered yes or no, it isn't ready. No fuzzy scores, no vibes.
  • External evaluators — the skill under test never judges itself. Deterministic checks first, LLM judges second.
  • Sacred regressions — a change that improves average score but breaks a sacred case is rejected. Period.
  • One change at a time — the optimizer proposes exactly one atomic change per experiment. No multi-variable confusion.
  • Baseline gating — if the base model already does the job without the skill, the skill gets flagged for obsolete review.
  • Model-aware — Haiku, Sonnet, and Opus are tested independently. Model variance is product reality, not noise.
  • Evidence-backed rollout — every ship/no-ship decision comes with a structured evidence trail.

Operator-Grade System Analysis

Architecture (Seven Layers)

┌─────────────────────────────────────────────────┐
│                   CLI / CI / API                 │  Layer 7: Surfaces
├─────────────────────────────────────────────────┤
│                 Evidence Layer                   │  Layer 6: Persistence
├─────────────────────────────────────────────────┤
│               Optimization Layer                 │  Layer 5: Experiments
├─────────────────────────────────────────────────┤
│                Judgment Layer                    │  Layer 4: Scoring
├─────────────────────────────────────────────────┤
│              Observation Layer                   │  Layer 3: Capture
├─────────────────────────────────────────────────┤
│               Execution Layer                    │  Layer 2: Harness
├─────────────────────────────────────────────────┤
│                  Spec Layer                      │  Layer 1: Contracts
└─────────────────────────────────────────────────┘
LayerResponsibilityKey Entities
SpecHuman-authored YAML eval contracts, criteria, test caseseval_specs, criteria, test_cases
ExecutionRuns skills against trigger, functional, regression, adversarial, baseline casesruns, skill_versions
ObservationCaptures outputs, artifacts, cost, latency, timing, observed outcomesobserved_outcomes, outputs
JudgmentDeterministic checks first, external LLM judges second, calibration, disagreement handlingcriterion_results
OptimizationFailure clustering, weakest-criterion targeting, single atomic changes, accept/reject/revertexperiments
EvidenceStores runs, scores, artifacts, diffs, regressions, baselines, launch reportsregressions, baselines, launch_reports
CLI/CI/APILocal author workflows, PR gating, team reporting, dashboard—

Epic Roadmap (10 Epics, Sequential)

#EpicScope
01Repo FoundationWorkspace skeleton, governance, CI
02Spec LayerYAML eval contracts, criteria schema, test case format
03Package IntegrityDeterministic structure/metadata validation
04Evidence LayerSQLite persistence, run lifecycle, evidence serialization
05Trigger HarnessRoster builder, trigger simulation, precision/recall
06Functional ExecutionSkill invocation, context injection, artifact capture
07Judgment LayerBinary judge engine, calibration, per-model matrix
08Regression/CLI/CIRegression comparison, baseline gating, score aggregation, CLI, PR gate
09OptimizerFailure clustering, one-change proposals, experiment runner
10Team ProductDashboard, eval packs, drift reevaluation, obsolete-review

Non-Negotiable Design Principles

  1. Criteria must be binary — yes or no, no gradients
  2. Evaluator is always separate — the skill never judges itself
  3. Observed behavior outranks claimed behavior — grade what happened, not what the skill says it does
  4. Regression tests are sacred — a regression on a sacred case blocks release regardless of average improvement
  5. One change at a time — optimizer proposes exactly one atomic change per experiment
  6. Blockers block release — a blocker failure cannot be averaged out
  7. Baseline value matters — if the naked model matches the skill, flag for obsolete review
  8. Model-aware testing is required — Haiku/Sonnet/Opus differences are product reality

Reference Library (32 files)

Self-contained library of templates, reference standards, agent patterns, and workflow diagrams under 000-docs/:

DirectoryContents
templates/skill-templates/6 SKILL.md structural patterns
templates/eval-schemas/Eval JSON schemas
references/skill-standards/AgentSkills.io spec, source-of-truth, frontmatter, validation rules
references/eval-patterns/Eval methodology, workflows, output patterns
references/agents/Grader, comparator, analyzer agent patterns
references/enterprise-standards/100-point rubric, production validator schema registry
references/drift-and-consistency/Drift categories, source-of-truth hierarchy
references/epic-workflows/10 ASCII workflow diagrams (one per epic)

Current Status

Phase: Epic 01 complete (repo foundation). Ready for Epic 02 (Spec Layer).

pnpm monorepo with four workspace packages (@j-rig/core, @j-rig/cli, @j-rig/db, @j-rig/dashboard), TypeScript baseline (tsup builds), quality guardrails (ESLint, Prettier, Vitest), and CI/CD workflows.

⚠️ Stub providers — output is NOT ground truth

The real Anthropic SDK adapter is not yet wired (tracked as iaj-stub-provider / PB-7 in the IEP Convergence Debt Plan). Every j-rig eval invocation currently runs against stub providers that emit synthetic outputs. The CLI refuses to run unless you explicitly opt in by setting J_RIG_ALLOW_STUB=1. When stub mode is active, a loud banner is emitted to stderr on every invocation.

Do not consume j-rig output as evidence of skill quality until the real provider lands. CI gates that ingest j-rig artifacts must refuse rows produced under stub mode.

Full discipline: STUB-PROVIDERS.md.


License

Apache 2.0 — see LICENSE and NOTICE for details.

Note: versions 0.x shipped under the MIT license. Starting with v1.0.0, the project is licensed under Apache 2.0. Any existing 0.x artifacts remain available under their original MIT terms; new releases (>= 1.0.0) are Apache 2.0.

Author

Jeremy Longshore — jeremylongshore · Intent Solutions

Ver o original no GitHub

Bom para

Relacionados