RAG + MCP Hyperparameter Tuning
Sudip-Pandit/rag-mcp-hyperparameter-tuning
Tunagem de 50 experimentos em RAG e MCP com código rodável offline
Ver no GitHubPlaybook executável para ajustar hiperparâmetros em sistemas RAG (chunk_size, similarity_threshold, top_k, rerankers) e agentes MCP (guardrails, memória, limites). Roda offline sem API keys, permitindo troca por modo live com OpenAI, Pinecone, Cohere ou Anthropic.
Instalação
npx convez-ai Sudip-Pandit/rag-mcp-hyperparameter-tuning1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Otimizar chunk_size e similarity_threshold em pipelines RAG
- Sintonizar estratégias de reranking e reescrita de queries
- Calibrar guardrails e limites de orçamento em agentes MCP
- Avaliar latência, custo e hit_rate em buscas semantic
Veredito

Ferramenta robusta e prática para quem trabalha com RAG em produção e precisa entender o real impacto de cada parâmetro. Código bem organizado com exemplos offline, ideal para experimentação antes de investir em APIs. Requer familiaridade com arquitetura RAG e MCP.
README
A tradução para o português deste README ainda está na fila.
RAG + MCP Hyperparameter Tuning
Runnable companion code for the article I Ran 50 Hyperparameter Experiments on a Production RAG System — Two Parameters Did 80% of the Work.
Every hyperparameter discussed in the article is a real, tunable knob here. The whole repo runs offline with zero API keys (local hashing embeddings, an in-memory vector store, a deterministic planner). Drop in OPENAI_API_KEY, PINECONE_API_KEY, COHERE_API_KEY, or ANTHROPIC_API_KEY to switch any stage to live mode.
Quick start
git clone https://github.com/Sudip-Pandit/rag-mcp-hyperparameter-tuning.git
cd rag-mcp-hparam-tuning
# Offline mode needs nothing installed. To run the whole tour:
python run_demo.py
That runs: synthetic data generation → indexing → a 20-config hyperparameter sweep → a ranked report → both case studies (HR Policy Bot and Support Ticket Agent).
For live API mode:
pip install -r requirements.txt
cp .env.example .env # fill in your keys
The two parameters that matter most
Per the article, similarity_threshold and chunk_size carry most of the quality gain. Both are first-class fields in rag_agent/config.py:RAGConfig and are swept in scripts/run_sweep.py.
Repository layout
rag-mcp-hparam-tuning/
├── rag_agent/ # Part 1: RAG tuning
│ ├── config.py master RAGConfig (every tunable parameter)
│ ├── chunker.py 1.1 token-aware chunking + cost estimator
│ ├── embedder.py 1.2 embedding model/dimension (OpenAI or local)
│ ├── retriever.py 1.3 top_k, metric, similarity_threshold + vector store
│ ├── reranker.py 1.4 cross-encoder / Cohere two-stage rerank
│ ├── query_rewriter.py 1.5 NONE/HyDE/multi-query/step-back/fusion
│ ├── generator.py 1.6 temperature/top_p/max_tokens/penalties
│ └── pipeline.py end-to-end RAG with cost + latency tracking
│
├── mcp_agent/ # Part 2: MCP tuning
│ ├── config.py tool gating + agent budgets + guardrails
│ ├── tools.py example ticket tools w/ risk levels
│ ├── router.py 2.1 confidence-gated tool selection
│ ├── guardrails.py 2.2/2.5 retry strategy + destructive-action gate
│ ├── memory.py 2.3 sliding memory window + summarization
│ └── agent.py bounded agent loop (max_steps, token budget)
│
├── evaluation/ # Part 1.7 / Part 3
│ ├── ragas_eval.py hit_rate, MRR, faithfulness triad
│ ├── latency_eval.py P50/P95/P99
│ ├── cost_eval.py per-query + projected monthly cost
│ └── report_generator.py ranked sweep report
│
├── shared/
│ ├── cost_tracker.py Part 3.1 cost model
│ ├── latency_tracker.py Part 3.3 stage timing
│ └── logging_config.py PII-redacting logger
│
├── scripts/
│ ├── generate_synthetic_data.py
│ ├── index_documents.py
│ ├── run_sweep.py Part 4.3 sweep (default 50 experiments)
│ └── benchmark.py Part 5 case studies, before/after
│
├── experiments/
│ ├── sweep_config.yaml canonical search space + A/B canary
│ └── results/ sweep_results.jsonl lands here
│
├── data/synthetic/ generated QA pairs + ground truth
├── run_demo.py one-command tour
├── requirements.txt
└── .env.example
Running pieces individually
# Each module has a __main__ smoke test:
python rag_agent/chunker.py
python -m rag_agent.pipeline
python -m mcp_agent.agent
# A bigger sweep (writes experiments/results/sweep_results.jsonl):
python scripts/run_sweep.py 50
python -m evaluation.report_generator
Production checklist (from the article)
RAG: set chunk_size by document type, keep chunk_overlap at 15–20%, always set similarity_threshold >= 0.6, add a reranker after validating baseline, temperature=0.0 for factual Q&A.
MCP: temperature=0.0 for tool calls, define stop_sequences, set max_steps and execution_budget_tokens, gate destructive tools behind human confirmation, log every tool call.
Live mode notes
- OpenAI: set
OPENAI_API_KEY; embeddings and generation switch automatically. - Pinecone: pass a real index object into
RAGPipeline(index=...); theretrieve_chunkscontract is identical. - Cohere reranker:
RerankerConfig(use_cohere=True)withCOHERE_API_KEY. - Anthropic MCP: set
ANTHROPIC_API_KEYto drive the agent with a live model (the offline planner stays available for deterministic tests).
Author
Written by Sudip P. Read the full walkthrough in the companion article, and follow along on Medium for more on production RAG, MCP, and AI engineering.
License
MIT. Use freely with attribution to the article.