convez.ai
AgenteAutomaçãoAvançado
model-collapse

Benchmark Agent Plugin for Claude Code

model-collapse/benchmark-agent-plugin

Benchmarks autônomos com análise cética, memory continuity e relatórios honestos

Ver no GitHub
InicialPluginScore · de 0 a 100

Agent autônomo que planeja, executa, monitora e analisa experimentos de desempenho em sistemas distribuídos com resultados reproduzíveis. Implementa protocolos de comparação justa, guarda-costas de segurança e análise cética com histórico persistente e detecção automática de conclusões tendenciosas.

Instalação

$npx convez-ai model-collapse/benchmark-agent-plugin

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
1
Forks
0
Acompanhando
0
Saúde
28%
Último commit
há 1 mês
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Comparação justa de otimizações de código em sistemas distribuídos
  • Automação de experimentos de desempenho com validação de conformidade
  • Análise cética de resultados com rastreamento de causalidade vs correlação

Veredito

Ferramenta especializada para benchmarking científico com rigor incomum: desafia hipóteses, aplica escrutínio maior a resultados bons, exige seção de falhas obrigatória. Ideal para quem precisa de conclusões confiáveis sobre desempenho, não marketing de performance.

README

A tradução para o português deste README ainda está na fila.

Benchmark Agent Plugin for Claude Code

An autonomous benchmark agent that plans, executes, monitors, and analyzes performance experiments on distributed systems. Produces reproducible results with honest, evidence-backed conclusions.

Install

# Add the marketplace
/plugin marketplace add model-collapse/benchmark-agent-plugin

# Install the plugin
/plugin install benchmark-agent@benchmark-tools

Or install directly from repo:

/plugin marketplace add model-collapse/benchmark-agent-plugin

Skills Included

SkillCommandPurpose
benchmark_agent/benchmark_agentBrain — plans, validates, generates scripts, analyzes results, manages history
benchmark_run/benchmark_runMuscle — launches nohup scripts, monitors via /loop, detects completion

How It Works

You: "benchmark force merge with the new eviction fix"
  → /benchmark_agent activates automatically
  → Loads prior run history and lessons
  → Challenges if request is misaligned with context
  → Produces run plan with comparison matrix
  → Generates execution/monitor/check scripts
  → You approve

You: "run it"
  → /benchmark_run launches nohup scripts
  → Starts /loop for periodic check-in
  → You can close your session

[Hours later, benchmark completes]
  → Loop detects completion signal
  → Triggers /benchmark_agent analyze
  → Skeptical analysis protocol runs
  → Report produced with mandatory "What Went Wrong" section
  → Results index updated with reliability tag

Core Principles

PrincipleMeaning
Fair ComparisonOnly the treatment variable changes between runs
Memory ContinuityEvery run inherits full context of prior runs
Safety FirstValidate hardware capacity before execution
Skeptical AnalysisGood results get more scrutiny, not less
Honest ReportingAll findings reported — especially inconvenient ones
Root Cause DepthNo guessing. Trace causation, not correlation

Key Features

  • Challenge Protocol — Agent pushes back on misaligned requests before wasting hours on confounded runs
  • Reliability Tags — Every run classified as RELIABLE / QUALIFIED / UNRELIABLE / EXPLORATORY
  • Session Resilience — Benchmark scripts survive session death via nohup; reconnect with /benchmark_run resume
  • Skeptical Analysis — Different protocols for good vs bad results (good results get MORE scrutiny)
  • Mandatory "What Went Wrong" Section — Reports without this section are rejected
  • Physical Artifact Verification — Catches logical vs physical file format mismatches
  • Phase Decomposition — Instruments per-phase to reveal true bottlenecks
  • Resource Budget — Computes worst-case before execution to prevent OOM

Data Directory

The plugin uses $BENCHMARK_HOME (defaults to ./benchmark_data/ in your project):

benchmark_data/
  results_index.md          — Single source of truth for all runs
  lessons.md                — Accumulated lessons (append-only)
  risks.md                  — Known fragilities and scaling limits
  runs/
    run_<date>_<slug>.md    — Individual run records
  plans/
    plan_<date>_<slug>.md   — Approved run plans
  scripts/
    execute_<id>.sh         — Generated execution scripts
    monitor_<id>.sh         — Generated resource monitors
    check_<id>.sh           — Generated status check scripts
  monitors/
    <id>_monitor.log        — CSV resource metrics
    <id>_events.log         — Phase transitions
    <id>.running            — Active run signal
    <id>.complete           — Completion signal
    <id>.failed             — Failure signal
  settings/
    <system>_settings.md    — Environment configs

Configuration

Set BENCHMARK_HOME environment variable to customize the data directory:

export BENCHMARK_HOME=/path/to/your/benchmark_data

Or leave unset — defaults to ./benchmark_data/ relative to your project root.

Origin

Built from lessons learned running force-merge/RSS/latency benchmarks on a 138M-doc SEISMIC sparse vector index across 3-node clusters. Every feature exists because its absence caused a real problem:

  • Comparison matrix: prevented by segment-count confound that nearly discarded a correct fix
  • Challenge protocol: prevented by a misaligned run that wasted 5+ hours
  • Reliability tags: prevented by citing an invalidated run as baseline
  • Physical artifact verification: prevented by 20 GB blind spot from compound file formats
  • Skeptical analysis: prevented by celebrating results that hid 6 bugs

See BENCHMARK_LESSONS_LEARNED.md for the full story.

License

Apache-2.0

Ver o original no GitHub

Bom para

Relacionados