convez.ai
Agente
ibrahimgunay

agentic_ai_eval

ibrahimgunay/agentic_ai_eval

Ver no GitHub
InicialPluginScore · de 0 a 100

A world-class evaluation pipeline for agentic AI features: understand the system, generate evals, and scaffold code from a description, sketch, or diagram. Backed by Claude with a deterministic offline mode.

Instalação

$npx convez-ai ibrahimgunay/agentic_ai_eval

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
57%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

README

A tradução para o português deste README ainda está na fila.


Agentic features are systems, not single model calls. A real feature is a graph of routers, planners, tool-use, retrieval, memory, sub-agents, and guardrails. Evaluating it means evaluating each component and the seams between them — with enough statistical rigor to make a ship/no-ship call you can defend. This pipeline operationalizes exactly that workflow.

Why it's different

Most "eval scripts" give you one number from one judge on a handful of cases. This gives you the workflow a frontier-lab eval team actually runs:

Typical eval scriptagentic-ai-eval
ModelHardcoded to one vendorAny of Anthropic / OpenAI / Gemini, auto-detected; fully offline with no key
CoverageA few promptsComponent-level + end-to-end evals derived from a failure-mode knowledge base
JudgeOne LLM callJury of judges (self-consistency + cross-model) with a measured uncertainty per verdict
HumansNoneHuman-in-the-loop review queue + judge↔human agreement (κ) and calibration
RigorA point estimateBootstrap confidence intervals, two-proportion significance tests for regressions
DataA printoutNormalized SQLite store, trend/A-B analytics, and an optional REST API
Output—A runnable agent scaffold + CI harness generated from the spec

The pipeline

  description / sketch / mermaid diagram
        │
        ▼  ingest        →  SystemSpec        (typed component graph + data flow)
        ▼  analyze       →  SystemAnalysis    (failure modes + prioritized risk register)
        ▼  generate      →  EvalSuite         (component + end-to-end evals, with graders)
        ▼  run           →  EvalReport        (jury + human grading, CIs, scorecard)
        ▼  persist       →  SQLite store      (trends, A/B regression gates, REST API)
        ▼  scaffold      →  agent.py + eval_harness.py + README
  1. Understand the system. Parse prose or a Mermaid flowchart into a typed SystemSpec: components (router, planner, tool_use, retrieval, memory, subagent, guardrail, generation, output_formatter, human_in_loop), data flow, constraints, and assumptions.
  2. Analyze failure modes. A curated knowledge base of per-component failure modes (misrouting, RAG hallucination, unsafe side-effecting tool calls, cross-session memory leakage, over-/under-refusal, …) yields a prioritized risk register with mitigations and the eval dimensions that matter.
  3. Generate evals. Targeted evals per component and end-to-end, each with appropriate graders: jury LLM-as-judge with rubrics, tool-trajectory matching, JSON-schema validation, exact/contains/regex, numeric thresholds (latency / cost / steps), and human graders.
  4. Run & grade. Execute the suite against your agent's traces. Each judged verdict carries a confidence interval; uncertain or borderline cases are routed to humans automatically.
  5. Persist & analyze. Every run lands in a SQL store you can trend over time, A/B with significance testing (a CI regression gate), and serve via REST.
  6. Scaffold code. Emit a runnable agent skeleton wired to a CI eval harness — go from a diagram to failing tests you can make pass.

Provider-agnostic by design

Set one key and the same pipeline runs unchanged:

export ANTHROPIC_API_KEY=...     # → claude-opus-4-8
# or
export OPENAI_API_KEY=...        # → gpt-4.1
# or
export GOOGLE_API_KEY=...        # → gemini-2.5-pro

The provider is auto-detected (or pin it with --provider / AGENTIC_EVAL_PROVIDER). With no key at all, every LLM stage degrades to a deterministic offline mode — heuristic parsing, knowledge-base analysis, template cases, neutral labelled judge — so the whole pipeline runs in CI and tests without spend. The core package depends on no vendor SDK; install only the provider you use.

Install

git clone https://github.com/ibrahimgunay/agentic_ai_eval.git
cd agentic_ai_eval

pip install -e .                 # offline-capable core, zero vendor SDKs
pip install -e ".[openai]"       # add a provider: openai | anthropic | gemini
pip install -e ".[all,dev]"      # everything + REST API + test tooling

cp .env.example .env             # add a provider key (optional)

Quick start

Python

from agentic_ai_eval import Pipeline

art = Pipeline().run(
    "A customer-support agent that routes intents, retrieves from docs (RAG), "
    "can issue refunds via a tool, and applies a safety guardrail.",
    name="support-agent",
)
r = art.report
print(f"{r.overall_score:.2f}  95% CI [{r.ci_low:.2f}, {r.ci_high:.2f}]  via {r.provider}")
print(r.by_dimension())

CLI

# Inspect the system + risk register
agentic-eval analyze examples/customer_support_agent.md

# Generate the eval suite
agentic-eval evals examples/customer_support_agent.md --out suite.json

# Full pipeline (dry run) -> ./runs/<name>/ + ingest into a SQL store
agentic-eval run examples/deep_research_agent.md --db eval.db

# A diagram works as input too
agentic-eval run examples/diagram.mmd

# Generate agent + harness code
agentic-eval scaffold examples/coding_agent.md --out generated/

Add --offline to any command to force deterministic mode, or --provider openai|anthropic|gemini to pin one.

Connecting your real agent

The eval runner takes a trace provider — a case_id -> Trace mapping or callable. That's the single seam where you plug in your agent:

from agentic_ai_eval.evals import run_suite
from agentic_ai_eval.schema import Trace

def my_traces(case_id: str) -> Trace:
    out = my_agent.run(case_input_for(case_id))   # your code
    return Trace(case_id=case_id, output=out.text, tool_calls=out.tools,
                 steps=out.steps, latency_ms=out.latency,
                 input_tokens=out.in_tok, output_tokens=out.out_tok)

report = run_suite(suite, my_traces)

The generated eval_harness.py shows the same wiring against the scaffolded agent.py.

Research-grade grading

A single judge call is a biased point estimate. The judge here is hardened the way frontier eval teams do it (see evals/judge.py):

from agentic_ai_eval.schema import Grader, GraderKind

# A 5-vote self-consistency jury, ensembled across two model families.
Grader(
    kind=GraderKind.LLM_JUDGE,
    rubric="Is every factual claim supported by the provided context?",
    jury_size=5,
    jury_models=["claude-opus-4-8", "gpt-4.1"],
)
  • Self-consistency — multiple sampled verdicts aggregated by median; their spread is a free uncertainty estimate surfaced on every GraderResult.
  • Cross-model jury — ensemble across models so no single model's idiosyncrasies dominate.
  • Calibration to humans — fit a monotone judge→human map on a gold set, with Cohen's κ and correlation, so you can quantify how far to trust the judge.

Every aggregate score ships with a bootstrap 95% CI, and run-to-run comparisons use a two-proportion z-test so regression gates fire on signal, not noise.

Human-in-the-loop

Put a person in the loop exactly where it pays off — and measure whether the automated judge has earned the right to stand in for them:

# 1. Export the cases that need a human (human graders + uncertain verdicts)
agentic-eval review export runs/deep_research_agent --out queue.csv

# 2. A reviewer fills in human_score / human_passed / notes (CSV or JSONL)

# 3. Merge verdicts back, re-aggregate, and print judge↔human agreement
agentic-eval review import queue.csv --run runs/deep_research_agent
#   → Judge ↔ human agreement: n=24 · κ=0.71 · r=0.83 · MAE=0.11

Cases are auto-routed to humans when a human grader is attached, when the jury disagreed, or when a score lands in the borderline band.

Data extraction & analytics

Every run is persisted to a normalized SQLite database — queryable by any BI tool, notebook, or the bundled REST API.

agentic-eval db ingest runs/support              # load a run
agentic-eval db runs --spec support              # list runs
agentic-eval db trend support                    # overall-score history
agentic-eval db compare <baseline> <candidate>   # significance-tested A/B (exits 1 on regression)
agentic-eval db query "SELECT dimension, AVG(score) FROM eval_results GROUP BY 1"
from agentic_ai_eval import EvalStore
from agentic_ai_eval.analytics import compare_runs

with EvalStore("eval.db") as store:
    rep = compare_runs(store, baseline_run, candidate_run)
    print("regressed dimensions:", [d.dimension for d in rep.regressed])

Serve it over HTTP for dashboards and CI bots (pip install '.[server]'):

agentic-eval serve --db eval.db
# GET /runs  /runs/{id}/evals  /specs/{spec}/trend  /compare?baseline=..&candidate=..

Example use cases

Specs modeled on real frontier agentic features ship in examples/:

SpecModels the pattern behind
deep_research_agent.mdOpenAI Deep Research · Anthropic Research · Gemini Deep Research
coding_agent.mdClaude Code · OpenAI Codex · SWE-bench agents
computer_use_agent.mdAnthropic Computer Use · OpenAI Operator · Google Mariner
customer_support_agent.mdRAG + tool-use + guardrail support agent

Architecture

ModuleResponsibility
schema.pyTyped contract for every stage (Pydantic).
providers/Vendor-neutral LLM backends (Anthropic / OpenAI / Gemini / offline).
llm.pyLLMClient facade: structured outputs + offline fallback.
ingest.pyDescription / Mermaid / JSON → SystemSpec.
analyze.pyFailure-mode KB + risk register → SystemAnalysis.
evals/generate.pySystemSpec + analysis → EvalSuite.
evals/judge.pyCalibrated jury of judges + judge→human calibration.
evals/graders.pyGrader implementations (jury judge, human, deterministic).
evals/metrics.pyTrajectory / latency / cost / step metrics.
evals/runner.pyExecute suite against traces → EvalReport (+ CIs).
stats.pyBootstrap CIs, Wilson intervals, z-tests, Cohen's κ.
human.pyReview-queue export/import + judge↔human agreement.
store.pyNormalized SQLite results store.
analytics.pyTrends + significance-tested run comparison.
server.pyOptional FastAPI REST surface over the store.
scaffold.pyGenerate agent skeleton + CI harness.
report.pyMarkdown / JSON reporting.
pipeline.pyOrchestration + artifact persistence.
cli.pyagentic-eval command-line interface.

Tests

pytest        # fully offline, deterministic — no keys, no network, no spend

License

Apache-2.0

Ver o original no GitHub

Bom para

Relacionados