convez.ai
MCP ServerSegurançaAvançado
YorickLane

Epistemics MCP – Verificação de Capacidades de Agentes IA

YorickLane/epistemics-mcp

Verifica claims de capacidade de agentes IA antes do deploy

Ver no GitHub
InicialPluginScore · de 0 a 100

Servidor MCP que verifica claims de capacidade, versão e atualização antes de agentes IA os utilizarem. Institucionaliza 'zero-hop verify' como ferramentas callable, impedindo que agentes distribuam informações falsas sobre APIs, pacotes e funcionalidades.

Instalação

$npx convez-ai YorickLane/epistemics-mcp

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 2 semanas
Versão
v0.1.0
Issues abertas
0
Contribuidores
2

Casos de uso

  • Validar se uma API realmente suporta um feature antes do agente afirmar
  • Verificar versão e atualização de pacotes e dependências
  • Confirmar documentação e esquema de MCP vs API upstream
  • Detectar gaps entre descrições de tools e capacidades reais

Veredito

Ferramenta crítica para engenheiros construindo agents em produção. Resolve um problema real e repetido: agentes afirmarem capacidades inexistentes baseadas em schemas e vibes. A implementação é sólida (stdlib-first, extras opcionais) e o anchor case (Grok + X video) é convincente.

README

A tradução para o português deste README ainda está na fila.

epistemics-mcp

Verify capability claims before AI agents ship them. A Model Context Protocol server that turns "zero-hop verify" from an attention-layer rule into callable tools.

License: MIT Python MCP Status

Why

LLM agents (Claude, ChatGPT, Grok — all of them) ship capability claims based on schema descriptions, training-data recall, and the vibe of upstream docs. They get caught when reality diverges. The pattern repeats every session.

This MCP server exposes verification as tools the agent can call, not prompts it must remember.

The anchor case

2026-05-25, 02:00 BJ. I claimed a Grok MCP server couldn't transcribe X-embedded videos based on its tool description. Direct API probe to xAI Responses API with enable_video_understanding=true returned an actual video transcript — speaker name, on-screen content, verbatim quotes — in 26 seconds. The MCP server (a community fork) just hadn't wired the parameter. The xAI capability was always there. My claim was wrong, the conversation wasted a research detour, and the next agent will repeat it unless verification is institutionalized.

probe_api_endpoint makes the contradicting probe a single tool call. See examples/probe_grok_video.py for the exact replay that catches the wrong claim.

What it catches

  • API capability claims — "does endpoint X support feature Y?"
  • Version / freshness claims — "is package Z at the latest version?"
  • Documentation claims — "does this source URL actually say what I think?"
  • MCP-vs-API schema gaps (v0.2) — "does my MCP expose every parameter the upstream API actually supports?"

Install

The package splits into a lean stdlib core plus optional extras, so helper-only consumers don't pull the MCP server dependency tree:

InstallPullsUse when
pip install epistemics-mcpnothing (pure stdlib)you only need the anti_stale_directive prompt-layer helper
pip install epistemics-mcp[probe]httpx[socks]you use probe_api_endpoint as a Python library
pip install epistemics-mcp[server]mcp + httpx[socks]you want to run the MCP server

Behavior change (v0.1.1): the base install no longer pulls mcp / httpx / pydantic. To run the MCP server you now need the [server] extra. Existing installs that ran the server via a plain pip install -e . must reinstall as pip install -e '.[server]'. Motivated by dogfood (2026-05-25): a consumer that only needed the pure-stdlib helper was forced to pip install --no-deps to avoid the full server tree.

Quick start (MCP server)

git clone https://github.com/YorickLane/epistemics-mcp
cd epistemics-mcp
uv venv && uv pip install -e '.[server]'

# Register with Claude Code (per-user, stdio transport)
claude mcp add -s user -t stdio epistemics \
  $(pwd)/.venv/bin/python3 $(pwd)/server.py

Restart Claude Code; the probe_api_endpoint_tool is now available.

As a Python library

The pure-stdlib helper works on the base install (no extras):

from epistemics.tools.anti_stale import anti_stale_directive

directive = anti_stale_directive(
    today_iso="2026-05-25",
    ground_truth={"trade_status": "ACTIVE since 5/23"},
)
# embed `directive` verbatim into a downstream LLM prompt

The HTTP probe needs the [probe] extra (pip install epistemics-mcp[probe]):

from epistemics.tools.probe_api import probe_api_endpoint

verdict = probe_api_endpoint(
    method="POST",
    url="https://api.x.ai/v1/responses",
    headers={"Authorization": "Bearer ${XAI_API_KEY}"},
    body={
        "model": "grok-4.3",
        "input": [{"role": "user", "content": "Analyze the embedded video."}],
        "tools": [{
            "type": "x_search",
            "allowed_x_handles": ["someuser"],
            "enable_video_understanding": True,
        }],
    },
    expected_status=200,
    expected_response_contains=["video"],
    expected_response_lacks=["cannot transcribe", "do not support"],
)

print(verdict.verdict)              # "match" | "mismatch" | "error"
print(verdict.elapsed_ms)           # 26022
print(verdict.matched_substrings)   # ["video"]
print(verdict.response_excerpt)     # first 4000 chars

${VAR_NAME} placeholders in headers and body are resolved from os.environ at call time. Resolved secrets are never returned in the verdict — only the placeholder names are echoed back.

Usage pattern: broad probe first, narrow assert second

A common dogfood anti-pattern is "I think endpoint X supports Y, let me probe with the exact substring I expect." This still depends on stale caller knowledge of the upstream schema — probe_api_endpoint will only falsify the specific substring you guessed, not surface the real shape.

Better pattern:

  1. Broad probe — call the endpoint with no expected_response_contains, inspect response_excerpt to see the real shape.
  2. Narrow assert — re-probe with the exact strings observed from step 1.

The status-only probe in step 1 will emit a weak verification note in verdict.notes to remind you it's a discovery call, not a verification.

Tools (v0.1.0)

ToolWhatLLM needed?
✅ probe_api_endpointControlled HTTP request + assertion verdictNo
✅ anti_stale_directivePure string builder for "use these values, do not infer from training memory" prompt directive — anchors downstream LLM to caller-provided ground truth (date / state / version / pricing). Sits at prompt layer instead of tool layer.No

Roadmap

v0.1.x — Dogfood window. The killer feature is whether agents reach for this tool unprompted when about to ship a claim. Tracking real-world catches before expanding the toolset.

v0.2.0 — Three more tools (gated on dogfood signal):

ToolWhatLLM needed?
tool_param_diffKiller: diff an MCP server's inputSchema against the upstream vendor API's actual parameter surface; auto-detect MCP-server-vs-API gaps (would have caught the anchor case in 30 seconds)No
package_versionResolve latest from npm / pypi / crates / go / gh-releaseNo
capability_verifyProbe vendor API, judge if it does what claim saysYes (cheap model via OpenRouter)
freshness_checkHeuristic + judge: is this claim stale given current truth?Yes

v0.3.0+ — doc_fact_check (claim + sources → cited verdict), anchor-replay test harness (feed past [UNVERIFIED] claims, see which would have been caught).

Architecture

Single Python MCP server. Pure-mechanical tools (probe_api_endpoint, package_version) work with no LLM key. Judge tools (v0.2+) use an OpenAI-compatible client; defaults to OpenRouter, but any OpenAI-compatible endpoint works (Anthropic via OpenRouter, OpenAI direct, local llama.cpp).

epistemics-mcp/
├── server.py                      MCP server entry (FastMCP)
├── epistemics/
│   ├── tools/
│   │   ├── probe_api.py           Controlled HTTP request + assertion
│   │   ├── package_version.py     [v0.2] Registry latest version
│   │   ├── capability_verify.py   [v0.2] Vendor API + LLM judge
│   │   └── freshness_check.py     [v0.2] Claim staleness detector
│   └── judge.py                   [v0.2] LLM judge wrapper
├── examples/
│   └── probe_grok_video.py        Anchor replay (the case in "Why")
└── tests/                         Real-anchor replay suite

Design principles

  1. Mechanical-first. Verification that doesn't need an LLM is faster, cheaper, deterministic, and works offline. Reach for LLM judge only when the question is genuinely semantic.
  2. Secret-safe by default. Placeholder syntax ${VAR} resolves from env at call time; resolved values never leave the process. Only placeholder names are echoed back in the verdict.
  3. Honest verdicts. match / mismatch / error — no soft-language. mismatch lists exactly which substrings were missing or forbidden; error cites the failure mode (HTTP error / missing env var / timeout).
  4. No framework lock-in. Standalone Python library + thin MCP wrapper. Use without Claude Code if you want.

License

MIT — see LICENSE.

Acknowledgements

Built on Model Context Protocol by Anthropic. Inspired by the recurring pain of capability misclaims that ship before verification.

Ver o original no GitHub

Bom para

Relacionados