convez.ai
PluginAutomaçãoIntermediário
adriansanthosh77-dev

TokenSlim

adriansanthosh77-dev/TokenSlim

Reduz despesa de tokens em LLMs com cache semântico e memória compacta

Ver no GitHub
InicialPluginScore · de 0 a 100

Middleware Node.js que reduz consumo de tokens em workflows com LLMs através de Triple-Tier Memory (Anchor + Fact Sheet + Buffer) e cache semântico. Otimiza sessões longas com economia de 40-60% em cenários repetitivos, funcionando com qualquer modelo via API.

Instalação

$npx convez-ai adriansanthosh77-dev/TokenSlim

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
2
Forks
0
Acompanhando
0
Saúde
28%
Último commit
há 6 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Otimizar custo de sessões longas em chatbots e copilots
  • Reduzir overhead de prompts de sistema em workflows repetitivos
  • Cache inteligente de intenções em suporte técnico ou atendimento interno
  • Compactar histórico de conversas sem perder contexto crítico

Veredito

Ferramenta prática e honesta para reduzir custos reais em LLMs, com benchmarks credíveis e foco em sessões longas. Não é solução mágica, mas entrega ganhos comprovados em workflows com repetição e memória acumulada.

README

A tradução para o português deste README ainda está na fila.

Token Reduction Layer

Provider-agnostic Node.js middleware for reducing LLM spend in long-running chat and copilot workflows.

This repo is most useful when your product has one or more of these traits:

  • long sessions
  • repeated or similar user asks
  • large system prompts
  • persistent brand, product, or workflow context

It is less useful for one-off stateless prompts. The win comes from session efficiency, not magic prompt compression.

What it does

Token Reduction Layer combines four practical levers:

  1. Lean message cleanup
  2. Intent-aware semantic cache
  3. Compact memory for long threads
  4. Provider-agnostic request packaging

It works with any model you can call from JavaScript through modelCaller(request): OpenAI, Anthropic, Gemini, Groq, Ollama, Together, local models, or your own gateway.

Realistic savings

These are the honest ranges for the current approach:

WorkloadTypical savings
Short one-off prompts5-15%
Medium chat sessions15-35%
Long sessions with memory pressure20-45%
Repetitive support/internal workflows30-60%+

The biggest levers are usually:

  • shorter fixed prompt overhead
  • cache hit rate
  • memory compaction after history gets large

The smallest lever is usually word reordering inside a single short prompt.

Exact benchmark example

Run:

npm run bench

Current sample output using the gpt-4o tokenizer:

ScenarioBeforeAfterSaved
User prompt only443911.4%
Old JSON wrapper vs raw payload473917.0%
Old system prompt vs new system prompt1912586.9%
4-message request total31313955.6%
Long-thread memory compaction26523511.3%

Important: summarizing a very short thread can cost more than it saves. This is why memory compaction should only kick in after history is genuinely large enough.

How it works

1. Lean message cleanup

The compressor removes filler and shortens known long phrases without stripping critical literals.

Could you please kindly explain how machine learning and deep learning work?
explain how ML and DL work?

By default the layer now sends a raw compressed message, because raw is cheaper than wrapping every turn in JSON or intent tags.

Optional payload modes:

  • raw
  • tagged
  • json

2. Intent-aware semantic cache

The cache is safer than a plain text-similarity lookup. It can reject hits when:

  • intent changes
  • important literals conflict
  • quoted strings conflict
  • numbers conflict
  • keyword overlap is too weak

That makes it more believable for production use in workflows where colors, prices, names, and constraints matter.

3. Compact thread memory

Long chats get reshaped into:

[anchor]
original brief

[facts]
brand=Gadiva Hair Oil
primary_color=#C9A84C
secondary_color=#1A1A1A
tagline=Rooted in Nature, Defined by You.

[recent]
last N raw turns

This preserves durable context while reducing the cost of replaying the middle of the conversation.

4. Universal request handoff

The layer can either call the model directly through your own SDK wrapper or prepare the optimized request bundle first.

Quick start

git clone <repo>
cd token-reduction-layer
npm test
npm run bench

Any-model usage

const { TokenReductionLayer } = require('./src');

const trl = new TokenReductionLayer({
  provider: 'custom',
  model: 'gpt-4o-mini',
  modelCaller: async request => {
    return 'model reply here';
  },
});

const result = await trl.chat(
  'Could you please write three launch taglines for Gadiva Hair Oil?'
);

console.log(result.reply);
console.log(result.compressionStats);

OpenAI example

See examples/openai-compatible.js.

Anthropic example

Anthropic still works out of the box:

const { TokenReductionLayer } = require('./src');

const trl = new TokenReductionLayer({
  provider: 'anthropic',
  anthropicApiKey: process.env.ANTHROPIC_API_KEY,
  model: 'claude-sonnet-4-20250514',
});

const result = await trl.chat('Explain retrieval augmented generation.');
console.log(result.reply);

prepareRequest()

If you want full control over the final API call:

const trl = new TokenReductionLayer({
  provider: 'custom',
  modelCaller: async () => 'ok',
});

await trl.chat('Write a homepage headline');
const request = trl.prepareRequest();

request contains:

  • system
  • messages
  • model
  • maxTokens
  • provider

Configuration

new TokenReductionLayer({
  provider: 'custom',
  model: 'gpt-4o-mini',
  maxTokens: 1024,

  modelCaller: async request => {},
  summarizerCaller: async request => {},

  compressionEnabled: true,
  payloadMode: 'raw',                  // 'raw' | 'tagged' | 'json'
  expandContractions: false,
  customAbbreviations: {
    'retrieval augmented generation': 'RAG',
    'customer relationship management': 'CRM',
  },

  cacheEnabled: true,
  cacheThreshold: 0.55,
  cacheMinSharedKeywords: 2,
  cacheRequireIntentMatch: true,
  cacheMaxEntries: 500,
  cacheTtlMs: null,

  summarizeEnabled: true,
  summarizeEveryNTurns: 10,
  keepRecentMessages: 8,
  factFormat: 'compact',
})

Where the savings usually come from

LeverTypical effectNotes
Message cleanup5-20% per messageBest on chatty prompts
Raw payload instead of JSON5-15% vs wrapped payloadsSmall but steady
Semantic cache100% on hitsDepends on repetition
Memory compaction20-60% on long threadsStrongest once history grows
Combined session effect20-45% typicalHigher on repetitive workloads

Files

token-reduction-layer/
|-- src/
|   |-- constants.js
|   |-- compressor.js
|   |-- cache.js
|   |-- summarizer.js
|   `-- index.js
|-- examples/
|   |-- basic.js
|   |-- express-middleware.js
|   `-- openai-compatible.js
|-- scripts/
|   `-- benchmark.js
|-- tests/
|   |-- compressor.test.js
|   |-- cache.test.js
|   `-- index.test.js
|-- calculator.html
|-- package.json
`-- README.md

Tests

npm test

License

MIT

Ver o original no GitHub

Bom para

Relacionados