convez.ai
PluginDados & AnalyticsAvançado
sweNNN-svg

Turkish RAG Eval — Pipeline de RAG com Avaliação Automática

sweNNN-svg/turkish-rag-eval

RAG turco com avaliação automática, métricas RAGAS e observabilidade Grafana

Ver no GitHub
InicialPluginScore · de 0 a 100

Sistema de produção para RAG em documentos turcos que mede automaticamente a qualidade usando métricas RAGAS (faithfulness e relevância). Inclui persistência em PostgreSQL, dashboards Grafana em tempo real e comparação de estratégias de chunking (fixa vs. semântica) para otimizar trade-offs entre latência e acurácia.

Instalação

$npx convez-ai sweNNN-svg/turkish-rag-eval

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 7 meses
Versão
—
Issues abertas
0
Contribuidores
2

Casos de uso

  • Avaliar qualidade de retrieval em pipelines RAG de documentos em português/turco
  • Monitorar degradação de qualidade e detectar drift em produção
  • Comparar estratégias de chunking para otimizar latência vs. acurácia
  • Armazenar e analisar métricas de faithfulness e relevância ao longo do tempo

Veredito

Implementação sólida de RAG com foco em observabilidade e métricas quantificáveis. O projeto vai além do 'funciona' e mede *quanto bem* funciona — ideal para quem precisa de RAG confiável em produção. Arquitetura bem pensada (Qdrant, RAGAS, PostgreSQL + Grafana) e código documentado, mas requer conhecimento intermediário-avançado de LLMs, vector stores e DevOps.

README

A tradução para o português deste README ainda está na fila.

Turkish Enterprise RAG — Evaluation

Production-ready RAG pipeline for Turkish enterprise documents with automatic quality scoring via RAGAS metrics.

Why This Exists

Most RAG demos stop at "it works." This one measures how well it works.

Every query automatically scores:

  • Faithfulness — is the answer grounded in the source documents?
  • Answer Relevancy — does it actually answer the question?

Results are stored in PostgreSQL and visualized in Grafana in real time.

Architecture

PDF/DOCX Upload → Chunker (fixed | semantic) → Qdrant (vector store) ↓ User Query → LangChain Retriever → Anthropic Claude → RAGAS Evaluator ↓ PostgreSQL → Grafana

Quick Start

git clone https://github.com/sweNNN-svg/turkish-rag-eval
cd turkish-rag-eval
cp .env.example .env  # fill in your API keys
docker compose up

API: http://localhost:8001/docs
Grafana: http://localhost:3001 (admin / your password)

What It Tracks

MetricDescription
FaithfulnessAnswer grounded in retrieved chunks?
Answer RelevancyAnswer relevant to the question?
Latency (ms)End-to-end response time
Chunking Strategyfixed vs semantic comparison

Chunking Strategy Comparison

Same document, same questions, two strategies:

StrategyFaithfulnessAnswer RelevancyAvg Latency
Fixed0.330.82~3.8s
Semantic0.420.80~10s

Trade-off: Semantic chunking reduces hallucination (higher faithfulness) but is 3x slower. Fixed chunking answers questions more directly. Choice depends on use case — latency-sensitive vs accuracy-critical.

Tech Stack

LayerTechnology
APIFastAPI + Pydantic
OrchestrationLangChain
Vector StoreQdrant
LLMAnthropic Claude
EvaluationRAGAS
StoragePostgreSQL
ObservabilityGrafana
InfrastructureDocker Compose

Production Considerations

Why Qdrant over Chroma? Qdrant supports production-grade filtering, payload indexing, and horizontal scaling. Chroma is suitable for prototypes.

Why RAGAS for evaluation? RAGAS provides reference-free metrics — no ground truth labels needed. Faithfulness and answer relevancy can be computed on any query/response pair in production.

Why separate chunking strategies? Chunk boundaries directly affect retrieval quality. Fixed-size chunking is deterministic and fast; semantic chunking respects sentence boundaries and produces more coherent context windows. This project benchmarks both rather than assuming one is better.

Why store eval results in PostgreSQL? Enables longitudinal analysis — tracking quality degradation over time, comparing model versions, detecting retrieval drift. A Grafana alert can notify when faithfulness drops below threshold.

API Reference

POST /ingest

Upload a PDF or DOCX document. curl -X POST http://localhost:8001/ingest -F "file=@document.pdf" -F "collection_name=my_docs" -F "chunking_strategy=semantic"

POST /ask

Ask a question against ingested documents. curl -X POST http://localhost:8001/ask -H "Content-Type: application/json" -d '{"question": "Veri güvenliği politikası nedir?", "collection_name": "my_docs"}'

GET /health

curl http://localhost:8001/health

Project Structure

turkish-rag-eval/ ├── app/ │ ├── chain.py # LangChain RAG pipeline │ ├── config.py # Pydantic settings │ ├── database.py # PostgreSQL persistence │ ├── evaluator.py # RAGAS metrics │ ├── ingest.py # PDF/DOCX parser + chunking │ ├── main.py # FastAPI endpoints │ ├── models.py # Request/response schemas │ └── vectorstore.py # Qdrant integration ├── scripts/ │ └── compare_chunking.py # Fixed vs semantic benchmark ├── docker/ │ └── grafana/ # Dashboard provisioning ├── docker-compose.yml └── README.md

Ver o original no GitHub

Bom para

Relacionados