convez.ai
MCP ServerIntegração & APIsIntermediário
Shweta-Mishra-ai

TokenMizer

Shweta-Mishra-ai/tokenmizer

Memória em grafo para agentes IA — contexto e decisões que persistem

Ver no GitHub
BomPluginScore · de 0 a 100

Proxy local que atua como uma camada de memória persistente entre sua aplicação e qualquer LLM, mantendo contexto através de múltiplas sessões via grafo de conhecimento. Implementa compressão inteligente, cache semântico e checkpoint automático para maximizar eficiência de tokens.

Instalação

$npx convez-ai Shweta-Mishra-ai/tokenmizer

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
35
Forks
8
Acompanhando
1
Saúde
85%
Último commit
há 1 dia
Versão
v0.6.1
Issues abertas
1
Contribuidores
2

Casos de uso

  • Manter continuidade de contexto em sessões longas sem perda de informação
  • Reduzir consumo de tokens em 15-40% com compressão inteligente de entrada
  • Armazenar histórico completo de decisões e mudanças de projeto

Veredito

TokenMizer é uma solução técnica robusta para o problema real de context window em workflows de IA. A abordagem de proxy local com pipeline de 6 camadas é bem arquitetada, mas requer integração cuidadosa e monitoramento de overhead.

README

A tradução para o português deste README ainda está na fila.


The Problem

Every AI session has a context limit. When you hit it:

  • The model forgets every decision, rationale, and context built over hours
  • You waste 10–30 minutes re-explaining the project every new session
  • Large files (CSV, PDF, Excel) eat your entire token budget instantly

How TokenMizer Solves It

TokenMizer is a local proxy between your app and any LLM. Every request goes through a pipeline that builds a live knowledge graph, compresses inputs, caches responses, and auto-checkpoints before context runs out.

Your App  →  TokenMizer (:8000)  →  Claude / GPT / Gemini / any LLM
                    │
          ┌─────────┴──────────────┐
          │   6-Layer Pipeline     │
          │   L0  File Intel       │  CSV/PDF/Excel → schema + sample
          │   L1  Compression      │  15–40% input reduction
          │   L2  Output Trim      │  5–15% output reduction
          │   L3  Semantic Cache   │  100% on repeated queries
          │   L4  Graph Memory     │  session continuity
          │   L5  Prompt Cache     │  90% on repeated system prompts
          └────────────────────────┘

Architecture

Decision Memory — 4-State Model

StatusMeaningIn Resume
🟢 ACTIVECurrent — in effect✅ Always
🟡 SUPERSEDEDReplaced by newer decision⚠️ 7 days
🔴 INVALIDATEDExplicitly wrong/cancelled⚠️ Always (warning)
⬜ ARCHIVEDOld but valid, not relevant❌ Never

History is never deleted. "Why did we switch from React to Next.js?" — always answerable.


Quick Start

1. Install

# Recommended
pip install "tokenmizer[anthropic,cache]"

# All providers
pip install "tokenmizer[anthropic,openai,gemini,cohere,cache]"

# No key? Use Ollama (free, local)
brew install ollama && ollama pull llama3
pip install tokenmizer

2. Set your API key

export TOKENMIZER_ANTHROPIC_API_KEY=sk-ant-...
# or: TOKENMIZER_OPENAI_API_KEY, TOKENMIZER_GEMINI_API_KEY, etc.

3. Start

tokenmizer serve
# → Proxy:     http://localhost:8000/v1/chat/completions
# → Dashboard: http://localhost:8000
# → API docs:  http://localhost:8000/docs

4. Use — change one line

from openai import OpenAI

client = OpenAI(
    api_key="your-key",
    base_url="http://localhost:8000/v1",  # ← only this changes
)

response = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Let's build an auth service"}],
    extra_body={"session_id": "my-project"},  # enables graph memory
)

Claude Code Integration

Option A — Plugin (recommended)

# Add TokenMizer as a plugin marketplace
/plugin marketplace add Shweta-Mishra-ai/tokenmizer

# Install
/plugin install tokenmizer@Shweta-Mishra-ai/tokenmizer

Then use skills directly:

/tokenmizer:checkpoint my-project      → save session to graph memory
/tokenmizer:resume my-project          → load previous session (300 tokens)
/tokenmizer:resume my-project full     → full 600-token context
/tokenmizer:analyze /data/sales.csv    → analyze file (99% token savings)
/tokenmizer:stats                      → token savings report

Option B — MCP server

Add to ~/.claude/settings.json:

{
  "mcpServers": {
    "tokenmizer": {
      "command": "python",
      "args": ["-m", "tokenmizer.mcp.server"],
      "env": { "TOKENMIZER_URL": "http://localhost:8000" }
    }
  }
}

Other Tools

Cursor / Continue.dev / any OpenAI-compatible tool:

API Base URL:  http://localhost:8000/v1

Session Resume

tokenmizer checkpoint my-project
tokenmizer resume my-project
Goal: Build FastAPI auth service with JWT + PostgreSQL
Done: Project setup | User model | Login endpoint | Fix 422 | 18 tests passing
In progress: Refresh token rotation
Decided: PostgreSQL (concurrent writes) | bcrypt | Redis for refresh tokens
Changed: ~~React~~ → Next.js (better SEO)
Files: api/auth.py, api/models.py, config.py
Continue: Implement token refresh endpoint

247 tokens replaces 25,000+ tokens of conversation history.


File Intelligence

from tokenmizer.filters.file_intelligence import FileIntelligence

fi = FileIntelligence()
result = fi.process(open("sales.csv","rb").read(), "sales.csv",
                    token_budget=500, query="which regions underperforming")
# 412,000 tokens → 447 tokens  (99.9% saved)
FileSavings
CSV (50k rows)99.9%
PDF (200 pages)98.8%
Excel (10 sheets)99.7%
JSON (1k items)95%

Works Alongside Caveman & CodeBurn

TokenMizer complements — does not replace — these tools:

ToolWhat it does
CavemanOutput tokens shorter (~65%)
CodeBurnInput context trimming
TokenMizerGraph memory + resume + file intelligence + cache

Tip: If using Caveman, set terse_output: enabled: false in tokenmizer.yaml to avoid conflicting system prompts.


Supported Providers

ProviderEnv var
Anthropic (Claude)TOKENMIZER_ANTHROPIC_API_KEY
OpenAITOKENMIZER_OPENAI_API_KEY
Google GeminiTOKENMIZER_GEMINI_API_KEY
DeepSeekTOKENMIZER_DEEPSEEK_API_KEY
MistralTOKENMIZER_MISTRAL_API_KEY
Grok (xAI)TOKENMIZER_GROK_API_KEY
CohereTOKENMIZER_COHERE_API_KEY
OpenRouterTOKENMIZER_OPENROUTER_API_KEY
OllamaNo key — free, local

Configuration

# tokenmizer.yaml
provider: anthropic
default_model: claude-sonnet-4-6

graph_checkpoint:
  enabled: true
  trigger_at_percent: 0.85
  use_llm_extraction: false     # true = 80%+ recall, needs key (~$0.001/turn)

compression:
  enabled: true

cache:
  enabled: true
  max_size: 10000

state_backend: memory           # memory | redis (production)

All settings via env vars: TOKENMIZER_PROVIDER, TOKENMIZER_API_KEY, etc.


Docker

# Quick start
docker-compose up tokenmizer

# With Redis (production)
ANTHROPIC_API_KEY=sk-ant-... docker-compose up

# With proxy auth
TOKENMIZER_API_KEY=strong-key docker-compose up

API Reference

EndpointMethodDescription
/v1/chat/completionsPOSTOpenAI-compatible proxy
/api/resume/{id}GETGet resume context
/api/checkpointPOSTManual checkpoint
/api/decision/invalidatePOSTMark decision as invalid
/api/graph/{id}GETSession graph stats
/api/statsGETToken savings analytics
/healthGETHealth check
/docsGETSwagger UI

Security

  • API key auth — TOKENMIZER_API_KEY
  • Secret redaction before graph storage
  • Session-isolated cache (sensitive data never shared)
  • Prompt injection detection
  • CORS restricted

Benchmarks

python -m benchmarks.checkpoint_accuracy.runner
pytest tests/ -v

Heuristic (default): Task recall ~75% · Decision recall ~75% · File recall ~100%

With use_llm_extraction: true: Task recall ~85–90% · Decision recall ~85–90%


CLI

tokenmizer serve [--port 8000]
tokenmizer checkpoint <session-id>
tokenmizer resume <session-id> [--level standard|full|critical]
tokenmizer analyze <file> [query]
tokenmizer stats

Contributing

See CONTRIBUTING.md. Graph extraction contributions are the highest priority.

git clone https://github.com/Shweta-Mishra-ai/tokenmizer
pip install -e ".[dev]"
pytest tests/ -v && ruff check tokenmizer/

License

MIT © Shweta Mishra


Ver o original no GitHub

Bom para

Relacionados