convez.ai
PluginBusca & PesquisaAvançado
amangrewal1

Assistente RAG Multilíngue com Suporte a Code-Switching

amangrewal1/multilingual-rag-assistant

RAG multilíngue com code-switching, citações e avaliação rigorosa

Ver no GitHub
BomPluginScore · de 0 a 100

Plugin Python que implementa um assistente RAG (Retrieval-Augmented Generation) capaz de processar consultas em múltiplos idiomas e dialetos (inglês, espanhol, hindi, hinglish, spanglish, punjabi e punglish), com detecção automática de dialeto, gate de segurança, recuperação vetorial e geração fundamentada com citações inline. Inclui suíte completa de avaliações (golden set, regressão e red team) para garantir fidelidade, relevância e resistência a injeção de prompts.

Instalação

$npx convez-ai amangrewal1/multilingual-rag-assistant

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
100%
Último commit
há 7 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Sistemas de acessibilidade com suporte a múltiplos idiomas
  • Assistentes conversacionais para comunidades multilíngues
  • QA em conhecimento técnico (WCAG, screen readers) com detecção de consultas ambíguas ou inseguras
  • Validação de respostas grounded e detecção de alucinações em bases de conhecimento

Veredito

Solução robusta e bem-estruturada para RAG multilíngue com rigor em segurança e avaliação. Ideal para quem precisa servir usuários em idiomas misturados e exigir respostas com citações verificáveis; requer familiaridade com embeddings, LLMs e pipelines de avaliação.

README

A tradução para o português deste README ainda está na fila.

Multilingual Conversational AI for Accessibility

tests license

RAG assistant for code-switched queries. Answers grounded in an indexed knowledge base with inline citations, a safety gate for unsafe or ambiguous inputs, and an eval harness for golden / regression / red-team suites.

What it does

  • Dialects: English, Spanish, Hindi (Devanagari), Hinglish, Spanglish, Punjabi (Gurmukhi), Punglish.
  • Pipeline: detect dialect → safety gate → normalize-for-retrieval → top-k vector retrieval → grounded generation with [#n] citations → optional refusal with translated message.
  • Evals: golden set (grounded Q&A across dialects), regression (hallucination / ambiguity guards), red team (prompt injection, PII, medical advice, fabrication bait).

Eval metrics

The evals/ harness produces a JSON report with these metrics per suite:

MetricWhat it measuresTarget
faithfulness_meanAre claims in the answer supported by cited docs?≥ 0.85
answer_relevance_meanDoes the answer address what was asked?≥ 0.80
citation_precision_meanFraction of citations that point to the right doc≥ 0.90
refusal_accuracyFraction of refuse/allow decisions matching expected≥ 0.95
latency_p50_msMedian end-to-end latency per query< 2000 ms
latency_p95_ms95th-percentile latency< 5000 ms

Suite sizes

SuiteCasesPurpose
golden_set12Grounded Q&A across 7 dialects
regression7Known-failure guards (hallucination, ambiguity)
red_team8Prompt injection, PII extraction, medical advice, fabrication bait

Judge model is a stronger Claude tier than the assistant (see .env.example) to reduce self-grading bias. Runs require ANTHROPIC_API_KEY set.

Layout

src/              pipeline: retriever, translator, safety, rag, assistant, prompts
data/
  knowledge_base/ markdown docs indexed at startup (WCAG, screen readers, captions, etc.)
  evals/          golden_set.json, regression.json, red_team.json
evals/            harness + metrics + runner
cli.py            REPL and one-shot asking

Setup

pip install -r requirements.txt
cp .env.example .env           # then fill ANTHROPIC_API_KEY
python cli.py --rebuild-index  # build the vector index

Use

python cli.py --q "What contrast ratio does WCAG require for normal text?"
python cli.py --q "screen reader ke liye headings kyun important hain?"
python cli.py                  # REPL

Run evals

python -m evals.run_evals --suite golden
python -m evals.run_evals --suite regression
python -m evals.run_evals --suite red_team
python -m evals.run_evals --suite all

Reports land under data/eval_reports/<suite>.json.

Design notes

  • Retrieval embeddings use a multilingual MiniLM so Hindi / Spanish queries hit English docs without needing upfront translation, and translation is only used as a fallback when the heuristic detector is unsure.
  • Refusal messages are templated in English and translated into the user's dialect at response time, so refusals stay localized without duplicating copy per language.
  • Red-team eval treats prompt injection and fabrication separately from out-of-scope refusal: injection must be blocked, fabrication must be hedged, OOS must be declined politely.
  • Judge uses a stronger model than the assistant to reduce self-grading bias.
Ver o original no GitHub

Bom para

Relacionados