convez.ai
MCP ServerIntegração & APIsIntermediário
jbaham2

Claude Langfuse Plugin

jbaham2/claude-langfuse-plugin

Instrumentation, eval e monitoring de LLM apps com julgamento durável + dados ao vivo

Ver no GitHub
InicialPluginScore · de 0 a 100

MCP server que transforma Claude em especialista em Langfuse, oferecendo instrumentação, setup, avaliação, monitoramento e self-hosting. Integra 5 skills, 6 comandos, 3 agents e acesso direto aos dados do Langfuse com documentação viva que nunca fica obsoleta.

Instalação

$npx convez-ai jbaham2/claude-langfuse-plugin

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Configurar e validar instrumentação do Langfuse em aplicações Python/JS
  • Desenhar e executar estratégias de avaliação de modelos com datasets e experiments
  • Monitorar performance, custos e alertas de aplicações LLM em produção
  • Diagnosticar falhas de integração e revisar traces para identificar padrões de erro
  • Planejar e operar deployments self-hosted com sizing, backup e segurança

Veredito

Plugin maduro e bem-arquitetado que fornece acesso especializado ao Langfuse via MCP. A abordagem de "decisões duráveis + fatos vivos" reduz drift de documentação. Essencial para times que trabalham seriamente com observabilidade e avaliação de LLM applications.

README

A tradução para o português deste README ainda está na fila.

Claude Langfuse Plugin

A Claude Code plugin that turns Claude into an expert at Langfuse — instrumentation, setup, evaluation, monitoring, and self-hosting — for Langfuse Cloud and self-hosted deployments, across the Python and JS/TS SDKs and 50+ framework integrations.

It bundles 5 skills, 6 commands, 3 agents, 1 safety hook, and both Langfuse MCP servers, designed on one principle: distill durable judgment, fetch the facts live. The plugin carries the decisions and workflows (which eval method when, how to size a deployment, how to read an experiment); it fetches exact, version-sensitive code from the live Langfuse docs at runtime so it never goes stale.


Features

Skills (auto-activate when relevant)

SkillWhat it owns
langfuse (vendored, official)Instrumentation code, the langfuse-cli, live-docs access, prompt migration, judge calibration, CI/CD experiment gates, error analysis, SDK upgrades
langfuse-setupAdoption decisions (Cloud vs self-host, region), onboarding sequence, first-trace verification, production-readiness checklist
langfuse-evaluationEval strategy — methods, scores model, datasets/experiments, LLM-as-a-judge, code evaluators, human annotation, RAG/agent/multi-turn/external-pipeline evals, experiment interpretation
langfuse-monitoringDashboards, the Metrics API, score analytics, alerting (Spend-Alerts vs app cost)
langfuse-deploymentSelf-hosting: architecture & sizing, scaling, backups/upgrades/migrations, security & SSO

Commands (you invoke explicitly)

CommandDoes
/lf-setupGuides onboarding — deployment choice, keys, first trace, prod-readiness
/lf-evalDesigns/sets up an evaluation — method, scores, datasets/experiments, online/offline
/lf-monitorBuilds or queries monitoring — dashboards, metrics, alerting
/lf-deployPlans/operates a self-hosted deployment — sizing, scaling, backups, security
/lf-trace-reviewReviews traces to find, classify, and quantify failure modes
/lf-health-checkChecks a connection/project: auth, ingestion, recent activity

Agents (autonomous, delegated tasks)

AgentDoesAccess
eval-designerExplores your codebase and produces a concrete, app-specific evaluation planread + reasoning
setup-doctorDiagnoses why a setup isn't working (traces missing, auth, region mismatch, flush)read-only
trace-reviewerBuilds a quantified failure taxonomy from your traces and proposes fixes + regression casesread-only

Hook

A non-blocking PostToolUse check that warns if you accidentally hardcode a Langfuse secret key (sk-lf-…) in an edited file. High-signal, never blocks your edit.

MCP servers (.mcp.json)

  • langfuse-docs — public docs search/retrieval (no auth). 3 tools.
  • langfuse-data-platform — authenticated access to your project's prompts, traces, observations, scores, datasets, evaluators, metrics, and getHealth. 61 tools.

Installation

1. Prerequisites

  • Claude Code installed.
  • A Langfuse project (Cloud sign-up at cloud.langfuse.com, or self-hosted) and a project API key pair (pk-lf-… / sk-lf-…) from Project Settings → API Keys.

2. Install the plugin

Local (development / trying it out):

git clone https://github.com/jbaham2/claude-langfuse-plugin
claude --plugin-dir /path/to/claude-langfuse-plugin

Via a marketplace (shareable install):

/plugin marketplace add jbaham2/claude-langfuse-plugin
/plugin install claude-langfuse-plugin

(Exact marketplace commands can vary by Claude Code version — see the Claude Code plugin docs.)

3. Set credentials (never paste keys into chat)

Copy the template and fill it in:

cp skills/langfuse-setup/assets/.env.example .env

Set in your shell or .env:

export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."
export LANGFUSE_BASE_URL="https://cloud.langfuse.com"   # your region or self-host URL
export LANGFUSE_HOST="$LANGFUSE_BASE_URL"               # some tools read HOST instead

4. Enable the data-platform MCP

The authenticated MCP uses Basic Auth. Generate the token and export it before launching Claude Code (the value is interpolated into .mcp.json at load):

export LANGFUSE_MCP_AUTH="$(printf '%s:%s' "$LANGFUSE_PUBLIC_KEY" "$LANGFUSE_SECRET_KEY" | base64)"

Set the right region endpoint in .mcp.json (the url of langfuse-data-platform):

RegionURL
EU (default)https://cloud.langfuse.com/api/public/mcp
UShttps://us.cloud.langfuse.com/api/public/mcp
Japanhttps://jp.cloud.langfuse.com/api/public/mcp
HIPAAhttps://hipaa.cloud.langfuse.com/api/public/mcp
Self-hostedhttps://your-domain.com/api/public/mcp

⚠️ Read + write tools are enabled by default (it can create/delete prompts, scores, datasets, evaluators…). For read-only use, configure a client-side tool allowlist. The bundled diagnostic agents (setup-doctor, trace-reviewer) are read-only by instruction regardless.

The Docs MCP needs no auth — it works immediately.

5. Verify

In a Claude Code session inside your project:

/lf-health-check

You should get a green report (API reachable, authenticated, traces arriving). If not, the setup-doctor agent will diagnose it.


Quick start

  1. Instrument your app: /lf-setup trace my Python app that uses the OpenAI SDK → walks the deployment/region/keys decisions, then hands off to the official langfuse skill for current best-practice tracing code.
  2. Confirm it works: /lf-health-check → verifies traces are arriving.
  3. Evaluate before you ship a change: /lf-eval gate a prompt change to my support bot → designs a dataset + experiment + the right evaluators.

How-to guide

Just describe the goal — the relevant skill activates automatically. Use a command when you want to drive a specific workflow, or an agent for a heavier autonomous task.

I want to…Do this
Add Langfuse tracing to my app/lf-setup …, or just "add Langfuse tracing to this app" (the official skill supplies current code)
Decide Cloud vs self-host / pick a regionAsk — langfuse-setup covers the decision and the irreversible bits
Check my setup is healthy / traces aren't showing/lf-health-check, or "my Langfuse traces aren't showing up" → setup-doctor
Design an evaluation strategy/lf-eval …, or delegate to the eval-designer agent for an app-specific plan
Gate a prompt/model change before deploy"evaluate this change against a dataset before shipping" → datasets + experiments
Set up an LLM-as-a-judge (and trust it)Ask to set up a judge; calibration is handed to the official skill's judge-calibration workflow
Find what's failing in production/lf-trace-review …, or the trace-reviewer agent for a quantified failure taxonomy
Monitor cost / latency / quality/lf-monitor … (Metrics API for ad-hoc queries; specs out dashboards)
Set a spend alert/lf-monitor — and note: Spend Alerts = your Langfuse bill, not app LLM cost
Self-host, size, scale, back up, secure/lf-deploy …
Look up exact current SDK/API codeAny skill fetches it live (the Docs MCP / .md-append) rather than guessing

Use cases

These mirror Langfuse's own guides, reframed as "ask the plugin."

1. Systematic error analysis (→ Langfuse Error Analysis guide)

"Review the last 24h of traces for my support-bot and tell me what's actually going wrong."

The trace-reviewer agent pulls observations via the data-platform MCP (listObservations, filtered by name/level/time), classifies failures into a taxonomy (hallucination, wrong tool call, format, retrieval miss…), quantifies how often each occurs, and proposes which failing traces to turn into regression test cases — instead of you reading traces one by one.

2. Gate a change before deploy (→ Testing LLM Applications / Experiment Interpretation)

"I changed the system prompt. Make sure quality didn't regress before I ship."

langfuse-evaluation sets up an offline experiment: a dataset (seeded from real production traces), a task, and evaluators. After the run it walks the interpretation funnel — aggregate metrics → baseline diff → item-level regressions → annotate — so a "+4% average" can't hide a "−16% on critical cases."

3. Evaluate a RAG pipeline (→ RAG Observability & Evals)

"Help me evaluate my retrieval-augmented chatbot."

The plugin evaluates retrieval and generation independently — retrieval relevance / context precision & recall on the retrieval step, faithfulness / answer relevancy on the generation — so you learn which half is failing. It can use Ragas (reference-free, runs on production traces) and lets you iterate on chunking cheaply.

4. Evaluate an agent (→ Agent Evaluation)

"How do I evaluate my tool-using agent?"

The eval-designer agent reads your code and produces a plan covering trajectory + per-step + final answer (not just final-output accuracy): tool-selection scores at the observation level, end-to-end trajectory judgment at the trace level, and a phased rollout (manual trace inspection → online evaluators → dataset-based testing).

5. Calibrate an LLM judge (→ Calibrate your LLM-as-a-Judge)

"Is my hallucination judge actually trustworthy?"

langfuse-evaluation sets the judge up; calibration is delegated to the official skill's judge-calibration workflow — run the judge against a labeled dataset as an experiment and report accuracy (or a full confusion matrix / precision-recall) so you know whether to trust it before it gates anything.

6. Stand up production monitoring (→ Automated Evaluations + dashboards)

"Give me a production health and cost view, and alert me if cost spikes."

langfuse-monitoring specs the three standard dashboards (health, cost, quality), answers ad-hoc questions via the Metrics API (queryMetrics), and sets up alerting — clarifying that app-level cost alerts are a scheduled metrics check, while Langfuse Spend Alerts cover your Langfuse bill.

7. Self-host and scale (→ Self-hosting guides)

"Size a Helm deployment for ~5M traces/day and plan our backups."

/lf-deploy gives the components + minimum sizing, the throughput levers (scale workers by CPU, reduce ClickHouse ingestion reads, separate ingestion/UI), the per-component backup strategy, and the security checklist — deferring exact Helm/Terraform values to the live self-hosting docs.


How it's organized

The repo is one plugin plus an authoring workspace. The Claude Code loader only reads the plugin parts; the rest is documentation/build material (most of it git-ignored from a published package — see .gitignore).

claude-langfuse-plugin/
├── .claude-plugin/plugin.json   # manifest
├── .mcp.json                    # both Langfuse MCP servers
├── skills/                      # 5 skills (1 vendored + 4 authored)
├── commands/                    # 6 slash commands
├── agents/                      # 3 subagents
├── hooks/                       # non-blocking secret-key guard
├── README.md  ·  LICENSE
└── meta/                        # architecture docs (ROADMAP, DECISIONS, DISTILLATION-TRACKER)
    # workbench/ — authoring scratch (git-ignored)

Knowledge pipeline: fetch (live .md) → distill durable judgment → skills/<skill>/references/ (shipped) — with stale-prone facts referenced live at runtime. Rationale and full build history are in meta/DECISIONS.md and meta/ROADMAP.md.


Maintenance

  • Vendored skill: skills/langfuse/ is the official Langfuse skill, pinned to an upstream commit. Re-sync periodically using the procedure in skills/langfuse/VENDORED.md.
  • Self-describing MCP: the data-platform MCP evolves; the plugin inspects its tools live rather than hardcoding schemas, so it tolerates upstream changes.

Credits & acknowledgements

This plugin is built on Langfuse's open-source work and would not exist without it:

  • The official Langfuse Agent Skill — github.com/langfuse/skills (© Langfuse GmbH, MIT). It is vendored verbatim into skills/langfuse/ and provides the instrumentation, langfuse-cli, prompt-migration, judge-calibration, error-analysis, and CI/CD workflows. Our four skills are built to complement it, not replace it. Provenance and the upstream commit pin are recorded in skills/langfuse/VENDORED.md.
  • The Langfuse documentation, guides, and cookbooks — langfuse.com/docs and langfuse.com/guides — the source material distilled into the skills' references and the use-cases above.
  • The Langfuse MCP servers — the public docs MCP and the authenticated data-platform MCP.

Thank you to the Langfuse team and community.

License

MIT — see LICENSE. The vendored official Langfuse skill (skills/langfuse/) is also MIT, © Langfuse GmbH — see skills/langfuse/LICENSE.upstream and skills/langfuse/VENDORED.md.

Not affiliated with Langfuse; built on their open-source platform and docs.

Ver o original no GitHub

Bom para

Relacionados