Eval Studio
narmaku/eval-studio
Workspace completo para construir e avaliar sistemas de IA em produção
Ver no GitHubIDE interativa completa para construir, executar e iterar avaliações de sistemas de IA. Oferece workspace unificado com suporte a datasets, métricas de scoring, rubrics com assistência de IA e múltiplos modos de avaliação (Q&A, RAG, agents, comparação de modelos). Integra com qualquer provedor de LLM via LiteLLM e frameworks externos.
Instalação
npx convez-ai narmaku/eval-studio1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Avaliar qualidade de respostas em chatbots e assistentes
- Validar e benchmarkar pipelines RAG
- Comparar performance de múltiplos modelos de IA lado-a-lado
- Criar e refinar rubrics de avaliação com assistência de IA
- Executar avaliações de agentes autônomos com observação de tool-calls
Veredito

Solução robusta e bem arquitetada para avaliar sistemas de IA em escala. Combina interface intuitiva com flexibilidade técnica, permitindo tanto engenheiros quanto especialistas definirem e executarem avaliações complexas de forma sistemática. Stack moderno (React 19, FastAPI, LiteLLM) e design de adaptadores facilitam extensibilidade.
README
A tradução para o português deste README ainda está na fila.
eval-studio
The workspace for building, running, and improving AI evaluations — designed for engineers and subject-matter experts alike.
Overview
eval-studio goes beyond running AI evaluations. It is a complete workspace for building everything needed to evaluate AI systems successfully: datasets, scoring metrics, evaluation rubrics, and telemetry integrations — then using them seamlessly with any evaluation framework onboarded into the platform.
Whether you're iterating on a chatbot's response quality, validating a RAG pipeline, benchmarking model candidates, or scoring autonomous agents, eval-studio provides the tools to design evaluations, execute them at scale, and refine them with AI-assisted feedback — all from a single interactive UI accessible to both engineers and non-technical SMEs.
What you can do
- Build datasets — Import from any format (YAML, JSONL, JSON, CSV), auto-detect fields, map to eval-studio's schema, upload directories of files. Smart import handles lightspeed-evaluation, SQuAD, Alpaca, and custom formats.
- Design scoring rubrics — Create evaluation dimensions with AI assistance via rubric-kit. Generate rubrics from natural language, refine with feedback, compare scoring approaches.
- Configure LLM providers — Register any model endpoint (OpenAI-compatible, LiteLLM-backed). Manage API keys via environment variables, never stored directly.
- Run evaluations — Q&A benchmarks, RAG pipelines, interactive agent sessions, or side-by-side model arena. Live logs and progress streamed via WebSocket.
- Compare and iterate — Arena mode for head-to-head model comparison with visual leaderboards. Per-question drill-down across contestants.
- Plug in any evaluation framework — Adapter architecture supports onboarding external evaluation systems. lightspeed-evaluation is the first target integration.
Evaluation modes
| Mode | What it does |
|---|---|
| Q&A Evaluation | Run datasets against models with LLM-as-judge scoring |
| RAG Evaluation | Evaluate retrieval + generation with faithfulness and relevance metrics |
| Agent Chat | Live multi-turn conversations with tool-call observation and scoring |
| Model Arena | Same evaluation across multiple models side-by-side with leaderboard |
Tech Stack
- Frontend: React 19 + TypeScript, Vite, shadcn/ui + Tailwind CSS, Zustand
- Backend: FastAPI (Python 3.11+), SQLAlchemy 2.0, SQLite (MVP)
- LLM Access: LiteLLM proxy (100+ providers)
- Evaluation Design: rubric-kit for AI-assisted rubric generation and refinement
Development
# Backend
cd backend && uv sync && uv run uvicorn app.main:app --reload --port 8000
# Frontend
cd frontend && npm install && npm run dev
# Or via Make
make dev
License
Apache 2.0