Turkish RAG Eval — Pipeline de RAG com Avaliação Automática
sweNNN-svg/turkish-rag-eval
RAG turco com avaliação automática, métricas RAGAS e observabilidade Grafana
Ver no GitHubSistema de produção para RAG em documentos turcos que mede automaticamente a qualidade usando métricas RAGAS (faithfulness e relevância). Inclui persistência em PostgreSQL, dashboards Grafana em tempo real e comparação de estratégias de chunking (fixa vs. semântica) para otimizar trade-offs entre latência e acurácia.
Instalação
npx convez-ai sweNNN-svg/turkish-rag-eval1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Avaliar qualidade de retrieval em pipelines RAG de documentos em português/turco
- Monitorar degradação de qualidade e detectar drift em produção
- Comparar estratégias de chunking para otimizar latência vs. acurácia
- Armazenar e analisar métricas de faithfulness e relevância ao longo do tempo
Veredito

Implementação sólida de RAG com foco em observabilidade e métricas quantificáveis. O projeto vai além do 'funciona' e mede *quanto bem* funciona — ideal para quem precisa de RAG confiável em produção. Arquitetura bem pensada (Qdrant, RAGAS, PostgreSQL + Grafana) e código documentado, mas requer conhecimento intermediário-avançado de LLMs, vector stores e DevOps.
README
A tradução para o português deste README ainda está na fila.
Turkish Enterprise RAG — Evaluation
Production-ready RAG pipeline for Turkish enterprise documents with automatic quality scoring via RAGAS metrics.
Why This Exists
Most RAG demos stop at "it works." This one measures how well it works.
Every query automatically scores:
- Faithfulness — is the answer grounded in the source documents?
- Answer Relevancy — does it actually answer the question?
Results are stored in PostgreSQL and visualized in Grafana in real time.
Architecture
PDF/DOCX Upload → Chunker (fixed | semantic) → Qdrant (vector store) ↓ User Query → LangChain Retriever → Anthropic Claude → RAGAS Evaluator ↓ PostgreSQL → Grafana
Quick Start
git clone https://github.com/sweNNN-svg/turkish-rag-eval
cd turkish-rag-eval
cp .env.example .env # fill in your API keys
docker compose up
API: http://localhost:8001/docs
Grafana: http://localhost:3001 (admin / your password)
What It Tracks
| Metric | Description |
|---|---|
| Faithfulness | Answer grounded in retrieved chunks? |
| Answer Relevancy | Answer relevant to the question? |
| Latency (ms) | End-to-end response time |
| Chunking Strategy | fixed vs semantic comparison |
Chunking Strategy Comparison
Same document, same questions, two strategies:
| Strategy | Faithfulness | Answer Relevancy | Avg Latency |
|---|---|---|---|
| Fixed | 0.33 | 0.82 | ~3.8s |
| Semantic | 0.42 | 0.80 | ~10s |
Trade-off: Semantic chunking reduces hallucination (higher faithfulness) but is 3x slower. Fixed chunking answers questions more directly. Choice depends on use case — latency-sensitive vs accuracy-critical.
Tech Stack
| Layer | Technology |
|---|---|
| API | FastAPI + Pydantic |
| Orchestration | LangChain |
| Vector Store | Qdrant |
| LLM | Anthropic Claude |
| Evaluation | RAGAS |
| Storage | PostgreSQL |
| Observability | Grafana |
| Infrastructure | Docker Compose |
Production Considerations
Why Qdrant over Chroma? Qdrant supports production-grade filtering, payload indexing, and horizontal scaling. Chroma is suitable for prototypes.
Why RAGAS for evaluation? RAGAS provides reference-free metrics — no ground truth labels needed. Faithfulness and answer relevancy can be computed on any query/response pair in production.
Why separate chunking strategies? Chunk boundaries directly affect retrieval quality. Fixed-size chunking is deterministic and fast; semantic chunking respects sentence boundaries and produces more coherent context windows. This project benchmarks both rather than assuming one is better.
Why store eval results in PostgreSQL? Enables longitudinal analysis — tracking quality degradation over time, comparing model versions, detecting retrieval drift. A Grafana alert can notify when faithfulness drops below threshold.
API Reference
POST /ingest
Upload a PDF or DOCX document. curl -X POST http://localhost:8001/ingest -F "file=@document.pdf" -F "collection_name=my_docs" -F "chunking_strategy=semantic"
POST /ask
Ask a question against ingested documents. curl -X POST http://localhost:8001/ask -H "Content-Type: application/json" -d '{"question": "Veri güvenliği politikası nedir?", "collection_name": "my_docs"}'
GET /health
curl http://localhost:8001/health
Project Structure
turkish-rag-eval/ ├── app/ │ ├── chain.py # LangChain RAG pipeline │ ├── config.py # Pydantic settings │ ├── database.py # PostgreSQL persistence │ ├── evaluator.py # RAGAS metrics │ ├── ingest.py # PDF/DOCX parser + chunking │ ├── main.py # FastAPI endpoints │ ├── models.py # Request/response schemas │ └── vectorstore.py # Qdrant integration ├── scripts/ │ └── compare_chunking.py # Fixed vs semantic benchmark ├── docker/ │ └── grafana/ # Dashboard provisioning ├── docker-compose.yml └── README.md