convez.ai
MCP ServerDados & AnalyticsAvançado
narmaku

Eval Studio

narmaku/eval-studio

Workspace completo para construir e avaliar sistemas de IA em produção

Ver no GitHub
InicialPluginScore · de 0 a 100

IDE interativa completa para construir, executar e iterar avaliações de sistemas de IA. Oferece workspace unificado com suporte a datasets, métricas de scoring, rubrics com assistência de IA e múltiplos modos de avaliação (Q&A, RAG, agents, comparação de modelos). Integra com qualquer provedor de LLM via LiteLLM e frameworks externos.

Instalação

$npx convez-ai narmaku/eval-studio

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
2
Forks
2
Acompanhando
0
Saúde
57%
Último commit
há 3 meses
Versão
v0.1.2
Issues abertas
44
Contribuidores
1

Casos de uso

  • Avaliar qualidade de respostas em chatbots e assistentes
  • Validar e benchmarkar pipelines RAG
  • Comparar performance de múltiplos modelos de IA lado-a-lado
  • Criar e refinar rubrics de avaliação com assistência de IA
  • Executar avaliações de agentes autônomos com observação de tool-calls

Veredito

Solução robusta e bem arquitetada para avaliar sistemas de IA em escala. Combina interface intuitiva com flexibilidade técnica, permitindo tanto engenheiros quanto especialistas definirem e executarem avaliações complexas de forma sistemática. Stack moderno (React 19, FastAPI, LiteLLM) e design de adaptadores facilitam extensibilidade.

README

A tradução para o português deste README ainda está na fila.

eval-studio

The workspace for building, running, and improving AI evaluations — designed for engineers and subject-matter experts alike.

Overview

eval-studio goes beyond running AI evaluations. It is a complete workspace for building everything needed to evaluate AI systems successfully: datasets, scoring metrics, evaluation rubrics, and telemetry integrations — then using them seamlessly with any evaluation framework onboarded into the platform.

Whether you're iterating on a chatbot's response quality, validating a RAG pipeline, benchmarking model candidates, or scoring autonomous agents, eval-studio provides the tools to design evaluations, execute them at scale, and refine them with AI-assisted feedback — all from a single interactive UI accessible to both engineers and non-technical SMEs.

What you can do

  • Build datasets — Import from any format (YAML, JSONL, JSON, CSV), auto-detect fields, map to eval-studio's schema, upload directories of files. Smart import handles lightspeed-evaluation, SQuAD, Alpaca, and custom formats.
  • Design scoring rubrics — Create evaluation dimensions with AI assistance via rubric-kit. Generate rubrics from natural language, refine with feedback, compare scoring approaches.
  • Configure LLM providers — Register any model endpoint (OpenAI-compatible, LiteLLM-backed). Manage API keys via environment variables, never stored directly.
  • Run evaluations — Q&A benchmarks, RAG pipelines, interactive agent sessions, or side-by-side model arena. Live logs and progress streamed via WebSocket.
  • Compare and iterate — Arena mode for head-to-head model comparison with visual leaderboards. Per-question drill-down across contestants.
  • Plug in any evaluation framework — Adapter architecture supports onboarding external evaluation systems. lightspeed-evaluation is the first target integration.

Evaluation modes

ModeWhat it does
Q&A EvaluationRun datasets against models with LLM-as-judge scoring
RAG EvaluationEvaluate retrieval + generation with faithfulness and relevance metrics
Agent ChatLive multi-turn conversations with tool-call observation and scoring
Model ArenaSame evaluation across multiple models side-by-side with leaderboard

Tech Stack

  • Frontend: React 19 + TypeScript, Vite, shadcn/ui + Tailwind CSS, Zustand
  • Backend: FastAPI (Python 3.11+), SQLAlchemy 2.0, SQLite (MVP)
  • LLM Access: LiteLLM proxy (100+ providers)
  • Evaluation Design: rubric-kit for AI-assisted rubric generation and refinement

Development

# Backend
cd backend && uv sync && uv run uvicorn app.main:app --reload --port 8000

# Frontend
cd frontend && npm install && npm run dev

# Or via Make
make dev

License

Apache 2.0

Ver o original no GitHub

Bom para

Relacionados