convez.ai
Agente
Vexa-ai

local-llm-agent-bench

Vexa-ai/local-llm-agent-bench

Ver no GitHub
InicialPluginScore · de 0 a 100

Benchmark: Claude Code, Mistral Vibe & OpenCode CLI agents on local/Mistral LLMs via LiteLLM — harness, synthetic fixtures, raw runs, scorecard

Instalação

$npx convez-ai Vexa-ai/local-llm-agent-bench

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
37%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

README

A tradução para o português deste README ainda está na fila.

local-llm-agent benchmark — pack ei-agent (issue #22)

Proves a Claude-Code-style CLI agent works against a customer-local LLM stack (LiteLLM gateway → Mistral-family models) and quantifies the gap vs Claude — the 4-arm matrix from the pack scope. Scorecard: SCORECARD.md.

ArmCLIModelRoute
1Claude Codeclaude-opus-4-8 / claude-sonnet-4-6 (baseline)host credentials (ANTHROPIC_API_KEY or host claude login)
2Claude CodeMistralcontainerized Claude Code (own CLAUDE_CONFIG_DIR) → local LiteLLM container exposing the Anthropic /v1/messages route → Mistral La Plateforme
3Mistral VibeMistraldirect API, default config
4OpenCodeMistraldirect API (OpenAI-compatible), default config

Same suite, same prompts, default CLI configs, no per-arm tuning. Switching arms or models touches env/config only — there are no code forks of any CLI.

One entrypoint

# the Mistral key arrives via the sanctioned secrets seam — env-only, never on disk:
export MISTRAL_API_KEY="$(cd <vexa-secrets-checkout> && bin/secret no-prod/mistral.enc.env MISTRAL_API_KEY)"

ARM=<1-4> MODEL=<id> benchmarks/local-llm-agent/run.sh        # full T1+T2+T3 suite
ARM=2 MODEL=mistral-small-latest TASKS=t1 benchmarks/local-llm-agent/run.sh

Defaults: MODEL=mistral-small-latest (arms 2–4), MODEL=claude-opus-4-8 (arm 1), TASKS=t1,t2,t3. Results append to runs/arm<N>-<model>.jsonl (one machine-readable record per arm × model × task: scores, tool-loop metrics, tokens, wall-clock). Regenerate the scorecard tables with python3 lib/scorecard.py --write.

Task suite

  • T1 workspace-write — meeting-transcript fixture → per-meeting markdown note conforming to the frozen fixture schema (fixtures/schema/meeting-note.schema.md; the production schema belongs to ei-workspace). Scored: schema compliance, content accuracy vs reference key facts, unassisted completion.
  • T2 recall Q&A — questions over a multi-meeting markdown workspace; answers must cite source files. Scored: correctness, citation validity.
  • T3 tool-loop fidelity — deterministic multi-step file task; plus loop metrics (valid-tool-call rate, turns, flail) extracted from every arm's machine-readable event stream (Claude Code stream-json, Vibe --output json, OpenCode --format json).

Hard rules (enforced)

  • Synthetic fixtures only ever cross the external API boundary — every fixture under fixtures/ is fictional (Helios Dynamics), generated for this benchmark. No real meeting data.
  • No secrets on disk or in logs. MISTRAL_API_KEY is injected via the environment (sanctioned seam above) and forwarded to containers with docker run -e. The LiteLLM master key is a per-invocation random value for the local proxy only.
  • The benchmark stack (compose.yaml, project vexa-bench) is fully separate from the product compose stack; run.sh tears the proxy down after arm-2 runs (KEEP_LITELLM=1 to keep it).

Layout

run.sh             single entrypoint (ARM/MODEL/TASKS env)
compose.yaml       LiteLLM proxy (Anthropic /v1/messages route) — arm 2 only
litellm/           proxy model map (key comes from env)
docker/            per-arm agent container recipes (claude-code, vibe, opencode)
prompts/           the three task prompts (shared verbatim by every arm)
fixtures/          synthetic transcripts, frozen note schema, T2 workspace+questions, T3 data
lib/runner.py      orchestrates one arm over the suite, appends JSONL
lib/score.py       deterministic 0–1 scoring per task
lib/scorecard.py   aggregates runs/*.jsonl into the scorecard tables
runs/              committed machine-readable results (JSONL)
Ver o original no GitHub

Bom para

Relacionados