convez.ai
PluginBusca & PesquisaIntermediário
aashi2912

Chunk Lab

aashi2912/chunk-lab

Visualize como chunking muda resultados de RAG na mesma pergunta

Ver no GitHub
InicialPluginScore · de 0 a 100

Plataforma interativa que demonstra como diferentes estratégias de chunking impactam a qualidade da retrieval em RAG. Executa um pipeline completo: segmentação do documento (fixed-size, recursivo ou sentence-level), embedding com Voyage AI, busca por similaridade e geração de resposta com Claude. Perfeito para entender por que a estratégia de chunking é tão crítica quanto a escolha do banco de vetores.

Instalação

$npx convez-ai aashi2912/chunk-lab

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
28%
Último commit
há 6 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Ensinar estratégias de chunking para RAG
  • Comparar impacto de diferentes segmentações em retrieval
  • Prototipação de pipelines de embedding e busca

Veredito

Excelente ferramenta educacional que desmitifica RAG através da experimentação prática. O código é limpo, extensível e roda localmente sem atrito — ideal para quem quer entender como chunking afeta retrieval antes de escalar.

README

A tradução para o português deste README ainda está na fila.

Chunk Lab

Same document. Same question. Different chunking — different answer.

An interactive demo showing why chunking strategy matters as much as the vector database you pick. Built to demonstrate end-to-end RAG: real embeddings, real similarity search, real LLM answer generation.

What it does

You give Chunk Lab a document and a question. It runs a complete RAG pipeline:

  1. Chunks the document using your selected strategy (fixed-size, recursive, or sentence-level)
  2. Embeds each chunk into a vector using Voyage AI
  3. Searches by cosine similarity against the embedded query
  4. Generates an answer with Claude using only the top retrieved chunks

Then you switch strategies and rerun — and see how chunking choice changes everything downstream.


Stack

LayerTech
FrameworkNext.js 14 (App Router) + TypeScript
EmbeddingsVoyage AI (voyage-3-lite, 512-dim)
Vector storeIn-memory cosine similarity
LLMAnthropic Claude (claude-haiku-4-5)
StylingTailwind CSS
Deploy targetVercel

The vector store is intentionally in-memory. Pinecone, Weaviate, ChromaDB, and Qdrant all do the same fundamental thing — storage + similarity search. To swap one in, replace lib/vectorStore.ts. The rest of the app stays untouched.


Local setup

1. Clone and install

git clone https://github.com/<your-username>/chunk-lab.git
cd chunk-lab
npm install

2. Get API keys

3. Set env vars

Create .env.local in the project root:

ANTHROPIC_API_KEY=sk-ant-...
VOYAGE_API_KEY=pa-...

4. Run

npm run dev

Open http://localhost:3000.


Deploy to Vercel

1. Push to GitHub

git init
git add .
git commit -m "initial commit"
git branch -M main
git remote add origin https://github.com/<your-username>/chunk-lab.git
git push -u origin main

2. Import to Vercel

  • Go to vercel.com/new
  • Import your chunk-lab repo
  • Vercel auto-detects Next.js — no config needed

3. Add env vars in Vercel

In the import flow (or under Project → Settings → Environment Variables), add:

NameValue
ANTHROPIC_API_KEYyour Anthropic key
VOYAGE_API_KEYyour Voyage key

4. Deploy

Click Deploy. First build takes ~90 seconds. Subsequent deploys ship in ~30 seconds on every git push.


Project structure

chunk-lab/
├── app/
│   ├── api/chunk/route.ts      # The full RAG pipeline endpoint
│   ├── globals.css              # Tailwind + base styles
│   ├── layout.tsx               # Root layout + metadata
│   └── page.tsx                 # The UI
├── lib/
│   ├── chunking.ts              # 3 chunking strategies
│   ├── embeddings.ts            # Voyage AI embedding calls
│   ├── samples.ts               # Demo documents
│   └── vectorStore.ts           # In-memory cosine similarity
├── .env.example                 # Template for env vars
├── next.config.js
├── package.json
├── postcss.config.js
├── tailwind.config.js
├── tsconfig.json
└── README.md

Cost notes

For typical usage on the demo:

  • Voyage embeddings: ~free. Each run embeds ~10-20 chunks + 1 query. 200M tokens/month free tier is millions of runs.
  • Claude: ~$0.001 per run (Haiku is cheap, prompt is small).

If you embed Chunk Lab in a portfolio and it gets 1,000 visits with average 3 runs each → ~$3 in Claude costs. Budget accordingly.


Why no Pinecone or ChromaDB?

The honest answer: for demonstrating chunking effects, an in-memory cosine similarity is identical to what a vector DB does — minus the indexing, persistence, and clustering layers that matter at scale.

Adding Pinecone would have meant: another API key, another account, slower setup, and zero new insight for the demo's purpose. The decision is documented in lib/vectorStore.ts.

If you want to see Chunk Lab v2 with real Pinecone, swap the search() function. The interface stays the same.

Ver o original no GitHub

Bom para

Relacionados