convez.ai
MCP ServerDados & AnalyticsAvançado
bengtfrost

RAG BGE-M3 Tokenizer

bengtfrost/rag-bge-tokeniser

RAG soberano e local com tokenização exata e indexação paralela

Ver no GitHub
InicialPluginScore · de 0 a 100

Servidor MCP de alta performance para Recuperação Aumentada por Geração (RAG) local, otimizado para documentos legais e chunking exato por tokens. Integra embeddings BGE-M3, busca vetorial via SQLite-vec e orquestração via agente Goose com gateway LiteLLM.

Instalação

$npx convez-ai bengtfrost/rag-bge-tokeniser

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Análise e busca em grandes volumes de documentos legais
  • Construção de pipelines RAG locais com controle total sobre dados
  • Indexação paralela de múltiplos formatos (PDF, Markdown, RST, texto)

Veredito

Ferramenta robusta e bem arquitetada para RAG soberano em ambientes Linux. Excelente para quem precisa de processamento local com controle total, mas requer familiaridade com Goose, LiteLLM e configuração de infraestrutura.

README

A tradução para o português deste README ainda está na fila.

Th# RAG BGE-M3 Tokenizer - Sovereign AI Stack (v2.0.0)

A high-performance, local RAG (Retrieval-Augmented Generation) solution optimized for Fedora 44 (Sway) and Linux power users. This system utilizes exact token counting for chunking and parallel embedding processing, orchestrated by the Rust-powered Goose agent via a LiteLLM gateway.

Note: This project has migrated from mcp-cli to Goose to provide a stable, agentic environment without hardcoded session timeouts or unstable STDIO handling.

✨ Key Features (v2.0.0)

  • Goose Orchestration: Uses a stable, Rust-based agentic CLI that handles long-running "missions" (like large-scale indexing) without session timeouts.
  • LiteLLM Gateway: A central proxy that unifies Local LLMs (llama-server) and Cloud APIs (Mistral/Gemini) into a single OpenAI-compatible stream on Port 4000.
  • Multi-format Support: Ingest PDF, Markdown (.md), RST, and plain text.
  • Exact Tokenization: Powered by the BAAI/bge-m3 tokenizer to ensure 1:1 parity between chunking logic and embedding model constraints.
  • Parallel Indexing: Uses asyncio.Semaphore to process multiple files concurrently without blocking the server.
  • Sovereign Metadata: Direct SQL access to the vector database for structural analysis and statistics.

🏗 Architecture

graph TD
    A[goose Agent] -->|Port 4000| B(LiteLLM Gateway)
    B -->|Port 11434| C(Local LLM: Gemma/DeepSeek)
    B -->|Cloud API| D(Mistral/Gemini)
    A --- |MCP: STDIO| E{Sovereign Tools}
    E --> G1[rag_server: v2.0 RAG]
    E --> G2[sqlite-vec: Direct SQL]
    E --> G3[filesystem/fetch]

🛠 MCP Tools

ToolDescription
create_collectionCreate a new RAG collection.
ingest_fileIndex a file (Text, PDF, or Markdown) directly from disk.
ingest_directoryBatch index an entire directory with progress bars and ETA.
queryHybrid search using Vector ANN followed by a BGE-Reranking stage.
sqlite__read_queryPerform raw SQL queries on the metadata and vector stats.
delete_documentsSelective deletion or full collection purge.

🚀 Installation & Setup

1. Install Goose (The Orchestrator)

Install the binary directly for maximum performance:

curl -fsSL https://github.com/aaif-goose/goose/releases/download/stable/download_cli.sh | CONFIGURE=false bash

2. Configure the Stack

Update your ~/.config/goose/config.yaml to include your Sovereign tools. This configuration uses the openai provider type to redirect all traffic to your LiteLLM gateway.

GOOSE_TELEMETRY_ENABLED: false
active_provider: openai

providers:
  openai:
    type: openai
    base_url: http://localhost:4000/v1
    api_key: sk-unused

extensions:
  rag:
    enabled: true
    name: rag
    type: stdio
    cmd: /home/USER/.config/rag-bge-tokeniser/.venv/bin/python
    args:
      - /home/USER/.config/rag-bge-tokeniser/rag_server.py
  sqlite:
    enabled: true
    name: sqlite
    type: stdio
    cmd: uvx
    args:
      - mcp-server-sqlite
      - --db-path
      - /home/USER/.local/share/rag-bge-tokeniser/vectors.db
  filesystem:
    enabled: true
    name: filesystem
    type: stdio
    cmd: npx
    args:
      - "-y"
      - "@modelcontextprotocol/server-filesystem"
      - /home/USER/ai-docs
      - /home/USER/.code

3. Setup Aliases (~/.zshrc)

To ensure stability and bypass Linux keyring issues, we use environment variables to route traffic through the LiteLLM gateway:

# Gateway ensured start
_ensure_litellm() {
    if ! ss -tulpn | grep -q ":4000 "; then
        v_litellm && litellm -c ~/.config/litellm/config.yaml > /dev/null 2>&1 &
        while ! ss -tulpn | grep -q ":4000 "; do sleep 1; done
    fi
}

alias goose-local='_ensure_litellm && export OPENAI_API_KEY="sk-unused" && export OPENAI_BASE_URL="http://localhost:4000/v1" && GOOSE_MODEL=local goose session'
alias goose-smart='_ensure_litellm && export OPENAI_API_KEY="sk-unused" && export OPENAI_BASE_URL="http://localhost:4000/v1" && GOOSE_MODEL=mistral-large-latest goose session'

⚙️ Environment Variables (Optional)

You can tune the RAG performance via environment variables in the shell where the server is launched:

VariableDescriptionDefault
RAG_CHUNK_SIZEMaximum tokens per segment512
RAG_MAX_CONCURRENTMax files indexed in parallel3

📂 Project Structure

  • rag_server.py: Core MCP logic & Exact Tokenizer.
  • vectors.db: SQLite-vec database (Location: ~/.local/share/rag-bge-tokeniser/).
  • config.yaml: The primary Goose configuration file.

Author: Bengt Frost
License: MIT

Ver o original no GitHub

Bom para

Relacionados