Assistente RAG Multilíngue com Suporte a Code-Switching
amangrewal1/multilingual-rag-assistant
RAG multilíngue com code-switching, citações e avaliação rigorosa
Ver no GitHubPlugin Python que implementa um assistente RAG (Retrieval-Augmented Generation) capaz de processar consultas em múltiplos idiomas e dialetos (inglês, espanhol, hindi, hinglish, spanglish, punjabi e punglish), com detecção automática de dialeto, gate de segurança, recuperação vetorial e geração fundamentada com citações inline. Inclui suíte completa de avaliações (golden set, regressão e red team) para garantir fidelidade, relevância e resistência a injeção de prompts.
Instalação
npx convez-ai amangrewal1/multilingual-rag-assistant1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Sistemas de acessibilidade com suporte a múltiplos idiomas
- Assistentes conversacionais para comunidades multilíngues
- QA em conhecimento técnico (WCAG, screen readers) com detecção de consultas ambíguas ou inseguras
- Validação de respostas grounded e detecção de alucinações em bases de conhecimento
Veredito

Solução robusta e bem-estruturada para RAG multilíngue com rigor em segurança e avaliação. Ideal para quem precisa servir usuários em idiomas misturados e exigir respostas com citações verificáveis; requer familiaridade com embeddings, LLMs e pipelines de avaliação.
README
A tradução para o português deste README ainda está na fila.
Multilingual Conversational AI for Accessibility
RAG assistant for code-switched queries. Answers grounded in an indexed knowledge base with inline citations, a safety gate for unsafe or ambiguous inputs, and an eval harness for golden / regression / red-team suites.
What it does
- Dialects: English, Spanish, Hindi (Devanagari), Hinglish, Spanglish, Punjabi (Gurmukhi), Punglish.
- Pipeline: detect dialect → safety gate → normalize-for-retrieval → top-k vector retrieval → grounded generation with
[#n]citations → optional refusal with translated message. - Evals: golden set (grounded Q&A across dialects), regression (hallucination / ambiguity guards), red team (prompt injection, PII, medical advice, fabrication bait).
Eval metrics
The evals/ harness produces a JSON report with these metrics per suite:
| Metric | What it measures | Target |
|---|---|---|
faithfulness_mean | Are claims in the answer supported by cited docs? | ≥ 0.85 |
answer_relevance_mean | Does the answer address what was asked? | ≥ 0.80 |
citation_precision_mean | Fraction of citations that point to the right doc | ≥ 0.90 |
refusal_accuracy | Fraction of refuse/allow decisions matching expected | ≥ 0.95 |
latency_p50_ms | Median end-to-end latency per query | < 2000 ms |
latency_p95_ms | 95th-percentile latency | < 5000 ms |
Suite sizes
| Suite | Cases | Purpose |
|---|---|---|
golden_set | 12 | Grounded Q&A across 7 dialects |
regression | 7 | Known-failure guards (hallucination, ambiguity) |
red_team | 8 | Prompt injection, PII extraction, medical advice, fabrication bait |
Judge model is a stronger Claude tier than the assistant (see .env.example)
to reduce self-grading bias. Runs require ANTHROPIC_API_KEY set.
Layout
src/ pipeline: retriever, translator, safety, rag, assistant, prompts
data/
knowledge_base/ markdown docs indexed at startup (WCAG, screen readers, captions, etc.)
evals/ golden_set.json, regression.json, red_team.json
evals/ harness + metrics + runner
cli.py REPL and one-shot asking
Setup
pip install -r requirements.txt
cp .env.example .env # then fill ANTHROPIC_API_KEY
python cli.py --rebuild-index # build the vector index
Use
python cli.py --q "What contrast ratio does WCAG require for normal text?"
python cli.py --q "screen reader ke liye headings kyun important hain?"
python cli.py # REPL
Run evals
python -m evals.run_evals --suite golden
python -m evals.run_evals --suite regression
python -m evals.run_evals --suite red_team
python -m evals.run_evals --suite all
Reports land under data/eval_reports/<suite>.json.
Design notes
- Retrieval embeddings use a multilingual MiniLM so Hindi / Spanish queries hit English docs without needing upfront translation, and translation is only used as a fallback when the heuristic detector is unsure.
- Refusal messages are templated in English and translated into the user's dialect at response time, so refusals stay localized without duplicating copy per language.
- Red-team eval treats prompt injection and fabrication separately from out-of-scope refusal: injection must be blocked, fabrication must be hedged, OOS must be declined politely.
- Judge uses a stronger model than the assistant to reduce self-grading bias.