convez.ai
MCP Server
multivon-ai

multivon-mcp

multivon-ai/multivon-mcp

Ver no GitHub
InicialPluginScore · de 0 a 100

MCP server exposing multivon-eval + pdfhell as agent-callable tools. Drop into Claude Desktop, Cursor, Cline.

Instalação

$npx convez-ai multivon-ai/multivon-mcp

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
50%
Último commit
há 2 meses
Versão
v0.4.0
Issues abertas
0
Contribuidores
2

README

A tradução para o português deste README ainda está na fila.

multivon-mcp

PyPI Python License Downloads

Docs · Website · PyPI · multivon-eval (engine) · Changelog

These 22 tools cover what an autonomous eval agent needs to do its job: discover its own capabilities (eval_discover), normalize traces from any source (ingest_trace), and run calibrated evaluators against them. We put the framework behind an MCP boundary because we think eval ends up as specialized agents coordinating through the protocol rather than a SaaS dashboard.

An MCP server that gives AI coding agents direct access to evaluation tools. Drop into Claude Desktop, Claude Code, Cursor, Cline, or any Model Context Protocol–compatible agent.

When the agent is helping you build an LLM product, it can:

  • Score a RAG output for hallucination without you writing the scaffolding
  • Generate an adversarial PDF on demand to test your document AI
  • Run the full pdfhell mini-suite against a model and analyse the results
  • Produce a hash-chained audit pack for procurement diligence
  • Discover the full evaluation capability catalog as JSON

No copy-paste, and no asking the agent to figure out the SDK calls from python -c "..." one-liners.

Install

pip install multivon-mcp

Bare install pulls multivon-eval, pdfhell, and the MCP SDK. The provider SDKs (anthropic, openai, google-genai) come along too — bring your own API key in env.

Configure your agent

Claude Code

claude mcp add multivon --env ANTHROPIC_API_KEY=sk-ant-... -- multivon-mcp

(Or add the same mcpServers snippet below to a project-level .mcp.json — Claude Code does not read claude_desktop_config.json.)

Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "multivon": {
      "command": "multivon-mcp",
      "env": {
        "ANTHROPIC_API_KEY": "sk-ant-...",
        "OPENAI_API_KEY": "sk-proj-...",
        "GOOGLE_API_KEY": "AIza..."
      }
    }
  }
}

Restart Claude. The 22 tools become available; ask Claude "use multivon to evaluate this RAG output" and it figures out which tool to call.

Cursor

cursor.json or via Settings → MCP:

{ "mcpServers": { "multivon": { "command": "multivon-mcp" } } }

Cline / OpenCode / any MCP-compatible agent

Same shape — point at the multivon-mcp console script.

Local dev / debugging

From a clone of this repo:

mcp dev multivon_mcp/server.py

From a pip install (the file lives in site-packages, so resolve it):

mcp dev "$(python -c 'import multivon_mcp.server as s; print(s.__file__)')"

Opens the MCP Inspector UI in your browser. You can call any tool by name, see the JSON schemas, and watch the requests/responses.

The 22 tools

Discovery & document AI

ToolWhat it doesAPI key
eval_discoverFull machine-readable capability catalog (evaluators, traps, suites, calibration data, versions). Call first.No
pdfhell_makeGenerate one adversarial PDF + its answer key.No
pdfhell_runRun the pdfhell adversarial-PDF benchmark against a vision model. Returns pass rate, per-trap CIs, suite hash.Yes (vision)
eval_audit_packBuild a hash-chained, procurement-ready ZIP from a pdfhell run.No

RAG generation & retrieval

ToolWhat it doesAPI key
eval_faithfulnessQAG-graded faithfulness — is a RAG output grounded in the retrieved context?Yes
eval_hallucinationQAG-graded hallucination — does the output contain content NOT in context?Yes
eval_relevanceQAG-graded answer-vs-question relevance.Yes
eval_answer_accuracyQAG-graded semantic equivalence vs ground truth.Yes
eval_context_precisionRAG retrieval quality — are the retrieved chunks on-topic?Yes
eval_context_recallRAG retrieval completeness — does context contain enough info to answer?Yes

Safety, compliance, fairness

ToolWhat it doesAPI key
eval_toxicityQAG-graded toxicity / harmful-content detection.Yes
eval_biasQAG-graded bias across gender, race, politics, age, socioeconomic axes.Yes
eval_pii_detectionLocal-only regex scan for PII (GDPR / CCPA / PIPEDA / HIPAA packs).No
eval_schema_complianceValidate an LLM output against a JSON Schema.No

Agent & multimodal

ToolWhat it doesAPI key
eval_tool_call_accuracyDeterministic agent tool-call correctness. No LLM.No
eval_vqa_faithfulnessImage-grounded visual-QA faithfulness.Yes (vision)
eval_document_groundingMulti-page document-grounded faithfulness for document-AI agents.Yes (vision)

Agent traces. eval_tool_call_accuracy and the other agent-trace evaluators in multivon-eval (ToolArgumentAccuracy, ToolCallNecessity, TrajectoryEfficiency, AgentMemoryEval, PlanQuality, TaskCompletion, StepFaithfulness) take an agent_trace=[AgentStep(...)] plus expected_tool_calls=[...] on the case. Three-shape semantics matter: expected_tool_calls=None skips, [] asserts "no tools called", and [...] checks the trace contains the named calls in order. The MCP tool wraps this — pass the trace JSON via eval_ingest_trace first to normalize it from LangGraph / OpenAI Agents SDK / manual shapes. See the multivon-eval agent integrations for the source-of-truth tracer code.

Flexible scoring

ToolWhat it doesAPI key
eval_g_evalG-Eval holistic 0.0-1.0 scoring against a plain-English criterion.Yes
eval_custom_rubricScore against your own list of yes/no quality checks.Yes

Agent workflows (new in 0.3.0)

ToolWhat it doesAPI key
eval_compare_runsDiff two eval report JSONs — pass-rate delta, per-case regressions/improvements, McNemar p-value. Use after every fix to confirm it actually helped.No
eval_generate_casesGenerate N eval cases (input / expected_output / context) from a chunk of source text. Eliminates the cold-start when building a new suite.Yes (judge)
eval_ingest_traceConvert a JSON agent trace (LangGraph / OpenAI Agents / manual) into an EvalCase payload. Use to score trajectories your agent just executed.No

Example session

User: I just shipped a RAG endpoint. Can you check it for hallucinations?

Claude: I'll use multivon to evaluate it.
        [calls eval_discover to see what's available]
        [calls eval_faithfulness with your input/context/output]

→ score: 0.667 (passed: False), threshold: 0.9
  reason: 2/3 claims grounded
    ✓ "annual renewal" — supported by context
    ✓ "30-day notice" — supported by context
    ✗ "automatic upgrade" — NOT in context

Claude: Your RAG hallucinated the "automatic upgrade" detail. The context
        doesn't mention upgrades. I'd add a Hallucination evaluator to your CI
        gate, threshold ≥0.85, and re-prompt with explicit "only use facts
        from context" instructions.

Why these 22 tools (not all 44)

eval_discover returns the full 44-evaluator catalog, so the agent can always introspect everything. The 22 tools we expose directly are the ones agents actually call mid-edit:

  • RAG generation checks (faithfulness, hallucination, relevance, answer_accuracy)
  • RAG retrieval checks (context_precision, context_recall)
  • Safety / fairness guardrails (toxicity, bias)
  • Compliance (pii_detection, schema_compliance) — local-only, no API egress
  • Flexible scoring (g_eval, custom_rubric) for user-defined rubrics
  • Multimodal (vqa_faithfulness, document_grounding) for vision agents
  • Agent traces (tool_call_accuracy)
  • Document AI (pdfhell_run, pdfhell_make) — for any RAG-on-PDFs flow
  • Audit pack — when procurement is involved
  • Discover — meta-capability for planning
  • Agent workflows (compare_runs, generate_cases, ingest_trace) — the loop that turns one-shot scoring into iterative improvement

The three new 0.3.0 tools exist because evals pay off as a loop: generate a starting suite from your own docs (eval_generate_cases), run your agent over it, score the trace (eval_ingest_trace → eval_*), make a fix, then verify the fix improved things vs. the baseline (eval_compare_runs). Agents need that whole loop callable from within a conversation, or they fall back to ad-hoc judgment.

Exposing all 44 evaluators as MCP tools would bloat the agent's context window and overwhelm tool-selection. If you need an evaluator that's not directly exposed, the agent can still use multivon-eval as a library — eval_discover returns the import paths.

Dependencies

Hard pins (from pyproject.toml):

  • mcp[cli] >= 1.0 — official MCP Python SDK + the mcp dev inspector
  • multivon-eval >= 0.9.4 — the evaluator surface this wraps
  • pdfhell >= 0.1.0 — the adversarial-PDF benchmark this wraps

Recommended (effective floor for full feature parity):

  • multivon-eval >= 0.9.8 — pulls in the corrected calibrated-threshold logic from the 0.9.7 hotfix (which affects what eval_discover reports and any tool that surfaces benchmark numbers in its docstring), plus the bundled Claude Code skills + multivon-eval install-skills CLI from 0.9.8.
  • pdfhell >= 0.5.4 — pulls in the mini-v4 17-trap suite and the pdfhell.research autoresearch loop. The pdfhell_run --suite mini-v4 tool path assumes these are present.

The pyproject pins are kept loose so existing deployments don't break; pin the recommended floors yourself if you care about the corrected benchmark numbers or the new suites.

All Apache 2.0.

MCP server vs Claude Code skills vs eval-action — which one do I use?

multivon-eval ships three agent-facing surfaces. They overlap on what they call (the same evaluator catalog) but differ on where the agent lives.

SurfaceWhere the agent runsBest for
multivon-mcp (this repo)Any MCP-compatible client — Claude Desktop, Cursor, Cline, OpenCode, Claude CodeMid-edit scoring inside an IDE or chat app. Agent calls eval_faithfulness / eval_hallucination / etc. directly as tools.
Claude Code skills — eval-bootstrap, eval-audit, eval-explain (bundled in multivon-eval >= 0.9.8; install with multivon-eval install-skills)Claude Code onlyWorkflow-shaped tasks: scaffold an eval suite from a project description, pre-PR regression checks against a baseline, explaining why a particular evaluator was picked. The skills know how to call multivon-eval bootstrap / use compare_reports / etc. so the agent doesn't have to figure it out from docs.
eval-actionGitHub CIGate every PR on eval regressions automatically. Posts the Wilson-CI + McNemar verdict as a PR comment.

If you're building an LLM product and want the agent in your editor to score a RAG output without copy-pasting Python, use multivon-mcp. If you live in Claude Code and want the bootstrap → audit → explain loop wired up as native commands, use the bundled skills. For PR-time gating, use the GitHub Action. Most projects end up using more than one.

The Multivon ecosystem

Five public + one early-access package, all built on a shared evaluation engine:

RepoWhat it is
multivon-evalPython SDK — 44 evaluators + bootstrap CLI + multivon_eval.auto. The engine multivon-mcp wraps.
pdfhellAdversarial PDFs that break AI document readers — exposed here as pdfhell_run + pdfhell_make tools
multivon-mcp (you are here)MCP server — 22 tools from multivon-eval + pdfhell
eval-actionGitHub Action — runs the same evals on every PR
eval-framework-benchmarkReproducible head-to-head benchmark vs DeepEval + RAGAS
multivon-guard (early access)Local proxy that catches LLM coding agents leaking secrets / PII

License

Apache 2.0.

Citing

@software{multivon_mcp,
  title  = {multivon-mcp: MCP server exposing multivon-eval + pdfhell as agent-callable tools},
  author = {Multivon},
  year   = {2026},
  url    = {https://github.com/multivon-ai/multivon-mcp},
}
Ver o original no GitHub

Bom para

Relacionados