convez.ai
HookProdutividadeIntermediário
juninmd

Tokenix

juninmd/tokenix

Indexação semântica local que economiza 60-90% de tokens em IA

Ver no GitHub
BomPluginScore · de 0 a 100

CLI local em Rust que indexa repositórios por semântica e reduz o uso de tokens em LLMs em 60-90%. Executa 100% offline, sem dependências externas, e integra-se com Claude, Copilot, OpenAI e qualquer cliente MCP.

Instalação

$npx convez-ai juninmd/tokenix

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
14
Forks
1
Acompanhando
0
Saúde
71%
Último commit
há 1 dia
Versão
v1.3.0
Issues abertas
3
Contribuidores
3

Casos de uso

  • Reduzir custo de tokens ao usar agents de IA com grandes bases de código
  • Fornecer contexto compacto e relevante para Claude Code e GitHub Copilot
  • Integrar busca semântica offline em pipelines de desenvolvimento
  • Escanear secrets e aplicar filtros de saída antes de enviar para LLM

Veredito

Ferramenta sólida e bem executada para otimizar consumo de tokens em fluxos de código. Proposta de valor clara: menos ruído, mais economia. Suporte multiplataforma e integração com principais IDEs/agentes refletem maturidade técnica.

README

A tradução para o português deste README ainda está na fila.


tokenix is a local-first Rust CLI that helps AI coding agents understand a repository without dumping huge files into the prompt. It indexes your code, finds relevant chunks by meaning, returns compact file outlines, and can hook into AI tools to replace noisy reads and command output with smaller, more useful context. Works with Claude Code, GitHub Copilot, OpenAI Codex CLI, Gemini, and any MCP client. No Ollama or external server required.

Without tokenix:  Read(src/auth/middleware.rs) → 800 lines → ~2,400 tokens  (illustrative)
With tokenix:     tokenix read src/auth/middleware.rs → symbol outline → ~180 tokens

Savings depend on codebase size, AI behavior, and file sizes. Run tokenix gain to see your real numbers.


🖥 Interactive Dashboard

Run bare tokenix to open a terminal dashboard — six tabs, zero flags. ←/→ switch tabs, ↑/↓ move, q quits. Piped or non-TTY falls back to --help.


What Is tokenix?

AI coding agents often waste context on the wrong shape of information: entire files, long grep output, repeated build logs, and directory listings that are much larger than the useful signal inside them. tokenix is a context layer between the agent and your repository.

It does four jobs:

JobWhat tokenix doesWhy it matters
Index the repositoryWalks source files, splits them into symbol-aware chunks, and stores local embeddings in SQLiteThe agent can search by intent instead of opening files blindly
Read files compactlyReturns outlines, symbols, or line ranges instead of full files when possibleLarge files stop consuming thousands of unnecessary tokens
Intercept assistant toolsHooks into supported tools before large reads and rewrites noisy command outputOptimization happens automatically during normal AI sessions
Measure savingsLogs hook decisions and estimates token/cost reduction with tokenix gainYou can see whether it is actually helping on your codebase

tokenix is not a cloud service, not a vector database server, and not a replacement for your AI assistant. It is a local repository index plus a set of CLI and hook integrations that make the assistant's context smaller and more targeted.


⚡ Quick Install

Pre-built binary (recommended)

Every release ships a binary per platform. The version-less releases/latest/download/<asset> URL always resolves to the newest release, so you never pin a version.

PlatformAsset
Linux x86_64tokenix-linux-x86_64
Linux arm64tokenix-linux-aarch64
macOS x86_64tokenix-macos-x86_64
macOS arm64 (Apple Silicon)tokenix-macos-aarch64
Windows x86_64tokenix-windows-x86_64.exe
Windows x86_64 (GPU / DirectML)tokenix-windows-x86_64-directml.exe

Each release also publishes sha256sums.txt and SLSA build-provenance attestations — see SECURITY.md for how to verify a download.

macOS / Linux — swap the asset for your platform from the table:

curl -fsSL https://github.com/juninmd/tokenix/releases/latest/download/tokenix-linux-x86_64 -o tokenix
chmod +x tokenix && sudo mv tokenix /usr/local/bin/tokenix
tokenix doctor   # verify

Windows (PowerShell):

irm https://github.com/juninmd/tokenix/releases/latest/download/tokenix-windows-x86_64.exe -OutFile tokenix.exe

🤖 For AI agents / LLMs: prefer this prebuilt binary over cargo install (no Rust toolchain, no compile step). Always fetch the version-less URL https://github.com/juninmd/tokenix/releases/latest/download/tokenix-<platform> — it redirects to the latest release, so never hard-code a version. Detect the platform, download the matching asset from the table, mark it executable, then run tokenix doctor to confirm. The embedding model downloads automatically on first use.

From crates.io

cargo install tokenix --locked

From source

git clone https://github.com/juninmd/tokenix
cd tokenix
cargo install --path . --locked

Use --locked. It builds against the committed Cargo.lock; without it cargo install re-resolves dependencies and can pull an incompatible ureq into the ort-sys build script.

Requirements: a recent stable Rust toolchain (edition 2021). No Ollama, no Python, no external services.

The embedding model (nomic-embed-text-v1.5, ~130 MB) is downloaded automatically on first use and cached locally.


✨ Features

FeatureDescription
Semantic searchFind relevant code by meaning, not just keywords (tokenix query); cross-project with --link
Context artifactstokenix artifacts indexes non-code schemas, API docs, and specs via .tokenix/artifacts.json
Hybrid rankingFTS5 BM25 + vector cosine + RRF fusion for ranked results
Exact searchRegex/literal search over indexed content, no embedding (tokenix grep)
One-call task contexttokenix context combines semantic search, entry points, and compact outlines with strict budget modes (plan, debug, audit, security, review)
Graph-aware exploretokenix explore returns related symbols, relationship maps, and grouped source in one capped call
Repository packtokenix pack emits a budgeted, secret-safe repo map with changed-file packs, token maps, and safety reporting
Symbol graphtokenix symbols (--kind filters by symbol type), callers, callees, impact, flow, and cycles trace relationships, call-flow, and circular deps between indexed symbols
Import graphtokenix deps FILE shows file-level import dependencies (--reverse for importers, --transitive to follow the chain); external deps are tracked too
Int8-quantized embeddingsVectors are stored int8-quantized (4x smaller DB + daemon RAM, near-identical recall); legacy f32 indexes migrate automatically on the next tokenix index
JSON output--json on query, context, explore, read, symbols, callers, callees, deps (+ impact --format json) for scripts and agent pipelines
PC-friendly indexingtokenix index runs at below-normal OS priority by default so long index runs never starve the machine (--no-low-priority opts out)
Interactive HTML/Mermaid graphstokenix impact --format html|mermaid exports vis.js / Mermaid flowcharts; tokenix flow --format mermaid traces call flow
Cycle detectiontokenix cycles finds circular dependencies via Tarjan's strongly-connected components algorithm, dropping same-name (homonym) false positives and annotating each node with path:line
Token maptokenix tokenmap shows a directory tree with token counts per file/folder
Preference memorytokenix memory add/list stores global and project preferences in editable Markdown; context/explore include saved preferences
Dynamic language detectionMap custom file extensions to any built-in parser via a project .tokenix.toml — no recompile needed
Legacy VB6 + SQL sources.bas/.cls/.ctl/.frm/.vbp and .sql/.fnc/.trg/.pkg/.prc/.tab/.vw indexed with symbol-aware heuristic chunking (Sub/Function/Property, CREATE objects); UTF-16 SQL files decoded via BOM; binary files (e.g. .frx) skipped by a NUL sniff
Symbol-aware chunkingAST Tree-sitter parsers for Rust, Python, TypeScript, JavaScript, Go, C/C++
Multi-agent safe indexPID-based index lock prevents concurrent reindex; embeddings are committed per batch, so a killed index run resumes from the last completed batch
Smart file readerOutlines large files; supports --symbol and --lines reads
Hook-based interceptionPreToolUse intercepts large reads and rewrites noisy Bash commands before execution; thresholds tunable via [hook] in .tokenix.toml
Structural output compressionFuzzy grouping, compact git/cargo filters, NDJSON/JSON compaction, and ANSI/Emoji stripping
Local project filtersDrop .toml files in .tokenix/filters/ for project-scoped compression rules — highest priority over user and bundled filters
Output filters231 TOML output filters embedded in the binary (each homologated against 500 golden cases) — auto-applied to Bash output for uv, cargo, terraform, ansible, docker, kubectl, git, npm, pnpm, bun, deno, vite, pip, poetry, go, rust, helm, and more
Filter generationtokenix filter generate writes a TOML filter for a command; tokenix filter record captures real output for richer generation
GPU acceleration (opt-in)Build with --features directml (Windows) or --features cuda to run embeddings on GPU; GPU is used by default at runtime with automatic CPU fallback, or force CPU with --only-cpu
Environment diagnosticstokenix doctor reports the compiled backend, detected GPU, CUDA/cuDNN status, model cache, and daemon
Branch-aware indexingTOKENIX_BRANCH_AWARE=true isolates indexes per git branch
In-memory daemontokenix serve keeps model + index in RAM so repeated hook calls avoid reloading the model each invocation; tokenix daemon status|stop|restart manages it
Graceful fallbackExits 0 on errors — your AI session is never broken
Token budgetResults fit within a configurable token budget (default 1200)
Savings analyticstokenix gain — token summary, savings split by source (semantic index vs command filters), and by-tool histogram; --cost-estimate adds a per-model cost table (10 reference models across Anthropic / OpenAI / Google)
Slim MCP profiletokenix mcp --profile slim exposes 3 meta-tools instead of the full tool surface for hosts that support progressive discovery
MCP/prompt weight audittokenix prompt-audit --recommend --profile-impact connects to configured MCP servers, tokenizes tool schemas, and shows full-vs-slim MCP savings
Session audittokenix session-audit --cache-hygiene combines index freshness, hook history, MCP/tool weight, and prompt-cache stability risks
Conversation secret scantokenix scan-secrets — gitleaks-style credential scan of Claude / Gemini / Copilot / Antigravity conversation transcripts (no git); findings are always redacted, exits non-zero when any are found. Patterns live in TOML (assets/secret-rules/), extensible via ~/.tokenix/secret-rules/*.toml or <repo>/.tokenix/secret-rules/*.toml
Local-first, no dependenciesfastembed ONNX in-process — no Ollama, no server, no internet after first run

🔌 Supported AI Tools

ToolIntegration
Claude CodePreToolUse hooks in ~/.claude/settings.json or project .claude/settings.local.json
GitHub Copilot.github/copilot-instructions.md + VS Code-compatible .github/hooks/hooks.json
OpenAI Codex CLI~/.codex/hooks.json for PreToolUse Bash rewrites + optional shell helpers
Antigravitytokenix install-hook --tool antigravity — installs and validates a native PreToolUse plugin through agy plugin
Any MCP clienttokenix mcp — Model Context Protocol server over stdin/stdout (--tool mcp)

🚀 How It Works

tokenix has two modes:

  1. Manual mode: run tokenix query, tokenix read, tokenix context, etc. directly when you want compact context.
  2. Hook mode: install hooks so supported AI tools call tokenix automatically before large reads and before noisy Bash commands execute.

Output compression

tokenix includes structural output-filtering logic. It doesn't just truncate output; it understands the structure of common CLI tools.

  • Fuzzy grouping: collapses hundreds of Compiling… or Removing… lines into a single summary line.
  • Structural compaction: compacts pretty-printed JSON and NDJSON into single-line formats.
  • Signal preservation: keeps error messages and summaries even when the middle of a log is truncated.

🛠 Usage

1. Index your repository

cd my-project
tokenix index .

First run: the model (~130 MB) is downloaded automatically. Subsequent runs use the local cache.

2. Search

tokenix query "how does JWT validation work"      # semantic
tokenix query "database connection pooling" --budget 2000
tokenix grep "fn validate_token" --ignore-case    # exact regex/literal

3. One-call task context

tokenix context "fix login refresh token bug"
tokenix context "how does the indexer batch embeddings" --mode debug --budget 2000
tokenix context "review this auth change" --mode review --budget 1200
tokenix explore "run_hook hook_post compression" --budget 4000

4. Repository pack

tokenix pack --mode plan --budget 8000 --format markdown --token-map
tokenix pack --mode review --changed --budget 4000
tokenix pack --mode security --format json --output tokenix-security-pack.json

pack builds a stable repo map plus focused context for tools that cannot call tokenix directly. It respects the index, skips obvious secrets and build output, and supports plan, debug, audit, security, and review modes. Use --changed or --since <ref> for compact review packs.

5. Smart file reader

tokenix read src/auth/middleware.rs                           # symbol outline
tokenix read src/auth/middleware.rs --symbol validate_token   # targeted
tokenix read src/auth/middleware.rs --lines 45-80             # line range

6. Symbol graph & maps

tokenix symbols validate_token
tokenix symbols Token --kind struct          # filter by symbol kind
tokenix callers validate_token
tokenix callees run_hook
tokenix impact update_user --depth 2
tokenix impact update_user --format html --output update_user.html   # vis.js graph
tokenix deps src/indexer.rs                  # file-level import dependencies
tokenix deps src/store.rs --reverse          # who imports this file
tokenix deps src/daemon.rs --transitive      # follow the import chain
tokenix tokenmap                                                     # token tree
tokenix rebuild-graph   # recompute relationships without re-embedding

Most retrieval commands accept --json for machine-readable output:

tokenix query "jwt validation" --json | jq '.[0].path'
tokenix callers run_hook --json

7. Token savings analytics

tokenix gain                  # token summary + by-tool histogram
tokenix gain --history        # include per-call history
tokenix gain --cost-estimate  # add the per-model cost table
tokenix session-audit         # index + hook + MCP token-economy health

tokenix gain shows a BY SOURCE section splitting the savings between the semantic index (Read/Grep intercepts answered with outlines and index queries) and the command filters (Bash output compression), so you can see which half of tokenix is earning its keep. tokenix gain --cost-estimate prices the savings against 10 reference models across Anthropic, OpenAI, and Google. Prices are shown with their collection date (currently 2026-06-11) so the numbers stay auditable.

8. Audit MCP / tool weight

tokenix prompt-audit                  # every agent that has MCP config
tokenix prompt-audit --agent claude   # one agent (claude|codex|copilot|antigravity)
tokenix prompt-audit --json           # machine-readable
tokenix prompt-audit --recommend      # include practical reduction advice

Discovers the MCP servers configured for each agent, connects to each one live (initialize + tools/list), tokenizes the returned tool schemas, and warns when too many servers/tools inflate the effective system prompt. The base system prompt itself cannot be read by tools, so this is a relative bloat estimate: the native-tool baseline is approximate and HTTP/SSE servers are shown as unknown. Thresholds are overridable via TOKENIX_AUDIT_WARN_TOKENS, TOKENIX_AUDIT_WARN_SERVERS, and TOKENIX_AUDIT_WARN_TOOLS.

For MCP hosts that support progressive discovery, run tokenix mcp --profile slim. The slim profile advertises only tokenix_context, tokenix_search_tools, and tokenix_call, reducing tool-schema tokens while preserving access to the full tokenix capability set through the meta-tool path.

9. Benchmark

tokenix benchmark
tokenix benchmark --json

tokenix benchmark measures tokenix against a plain vanilla baseline using the actual index/search code — no external tools involved. It reports read-only token reduction on large files, targeted outline+symbol workflows, semantic search Hit@1/Hit@3, context homologation (vanilla full file vs tokenix budgeted context), and command-output compression. Scenarios span Rust/TS/Go/Python, SQLite vector search, and common command output (cargo, git, npm, docker compose); misses are included as measured. Pass --refresh-index to re-embed first, --cases FILE for project-specific cases, and --json for a machine-readable summary.

10. Scan conversations for exposed secrets

tokenix scan-secrets                          # all agents, redacted, exit 1 on hits
tokenix scan-secrets --group value            # one block per distinct secret
tokenix scan-secrets --group repo             # group by the repo it leaked from
tokenix scan-secrets --filter telegram        # filter rule/agent/file/value/repo/branch
tokenix scan-secrets --agent claude --json    # machine-readable
tokenix scan-secrets --filter aws --reveal    # print raw values (warns on stderr)

Like gitleaks --no-git, but it walks each AI agent's conversation transcripts (Claude ~/.claude/projects, Gemini ~/.gemini/tmp,history, Copilot ~/.copilot/session-state,logs, Antigravity ~/.gemini/antigravity) for pasted credentials. Every finding is redacted by default and attributed to the repository + git branch it was exposed in (from each Claude message's cwd/gitBranch, falling back to the project directory). Detection patterns are TOML [[rules]] in assets/secret-rules/, extensible without a rebuild via ~/.tokenix/secret-rules/*.toml or <repo>/.tokenix/secret-rules/*.toml.


🔧 Setup by Tool

Claude Code

tokenix install-hook --tool claude-code

Writes a PreToolUse hook to ~/.claude/settings.json (or .claude/settings.local.json with --local). Large reads, semantic greps, and noisy Bash commands are intercepted automatically — no changes to your prompts needed. hook-post (PostToolUse) remains a compatibility handler, not a default Claude install, because it cannot replace the original tool output.

GitHub Copilot

cd my-project
tokenix install-hook --tool copilot
git add .github/
git commit -m "chore: add tokenix context instructions"

Creates .github/copilot-instructions.md and .github/hooks/hooks.json.

OpenAI Codex CLI

tokenix install-hook --tool codex
# bash / zsh
echo 'source ~/.codex/tokenix-init.sh' >> ~/.bashrc
# PowerShell
echo '. ~/.codex/tokenix-init.ps1' >> $PROFILE

Then use tx-read and tx-query as shell helpers. On Windows this also installs ~/.codex/hooks.json and a PowerShell wrapper that forwards PreToolUse intercepts for Bash-like terminal tools (Bash, run_in_terminal) and normalizes grep_search to the same semantic path as Grep.

Antigravity

tokenix install-hook --tool antigravity
# workspace-only:
tokenix install-hook --tool antigravity --local

Global installation uses agy plugin install, validates the result, and stores it under ~/.gemini/config/plugins/tokenix/. Workspace installation writes .agents/plugins/tokenix/ and validates it with agy plugin validate. The native hook handles Antigravity's toolCall.name/args payload and returns decision: allow|deny; Bash savings use a PreToolUse overwrite. Antigravity PostToolUse cannot replace tool output, so tokenix does not install a no-op post hook.

All tools at once

tokenix install-hook --tool all

📖 Commands Reference

Run bare tokenix (or tokenix --help) for an audience-grouped command catalog with examples. The reference below mirrors that grouping: AI agent commands (the LLM/hooks drive these for token-lean retrieval) vs human commands (setup, ops & analytics you run yourself).

🤖 AI agent commands

CommandDescription
tokenix context TEXTOne-call task context: entry points, relevant source, compact outlines, strict budget modes
tokenix explore TEXTGraph-aware exploration: entry points, relationships, grouped source
tokenix query TEXTSemantic search over indexed chunks
tokenix grep PATTERNExact regex/literal search over indexed content (no embedding)
tokenix read FILESmart reader — outline for large files, full for small
tokenix symbols QUERYFind indexed symbols by name or path (--kind filters by symbol type)
tokenix callers SYMBOLShow symbols that call/reference a symbol
tokenix callees SYMBOLShow symbols called/referenced by a symbol
tokenix deps FILEFile-level import dependencies (--reverse, --transitive, --json)
tokenix impact SYMBOLBidirectional impact graph (--format html|mermaid for vis.js graph or Mermaid flowchart)
tokenix flow SYMBOLForward call-flow trace from a symbol (--depth, --format text|mermaid)
tokenix packBudgeted repo pack for non-hook AI tools (--mode/--profile, --changed, --token-map)
tokenix memory add TEXTSave a preference (--global or --project) for future context
tokenix memory listList global and project preferences
tokenix memory remove TEXTRemove preferences matching text
tokenix memory edit TEXTReplace preferences matching text

🧑 Human commands

CommandDescription
tokenix (no args)Open the interactive dashboard — Stats · Filters · Gain · Doctor · Tokenmap · Secrets tabs; piped/non-TTY falls back to help
tokenix filter (no args)Open the dashboard on the Filters tab; piped falls back to filter list
tokenix index [PATH]Index the repo at PATH (default .)
tokenix install-hookInstall assistant hook/instructions (default --tool all)
tokenix remove-hookRemove assistant hook/instructions (default --tool all)
tokenix install-binaryCopy the running executable to a per-user global bin dir (%LOCALAPPDATA%\tokenix\bin on Windows, ~/.local/bin on Linux/macOS) and ensure it is on PATH (Windows: user PATH updated automatically; Linux/macOS: prints the shell-profile line)
tokenix doctorDiagnose embedding backend, GPU availability, model cache, daemon, bundled filter inventory (filter + golden-case counts), active recording session, and user/local filter config (unknown semantic_filter.model, bad threshold)
tokenix serveStart the background embedding daemon (keeps model + index in RAM)
tokenix stopStop the background daemon
tokenix daemon status|stop|restartInspect (pid, port, uptime, model, cache RAM) or control the daemon
tokenix gainToken savings analytics with a by-source split — semantic index vs command filters (--cost-estimate adds a per-model cost table)
tokenix statsIndex statistics (files, chunks, tokens, age)
tokenix tokenmapDirectory tree map with token counts, heaviest paths first, plus a top-10 files summary (--format html supported)
tokenix benchmarkReproducible token-savings and retrieval-quality benchmark — vanilla vs tokenix (--json)
tokenix filter listShow top Bash commands by tokens wasted (no filter yet)
tokenix filter activeShow active user and bundled output filters
tokenix filter generate [CMD]AI-generate a TOML output filter for a command
tokenix filter record [CMD]Record real command output for richer filter generation
tokenix prompt-auditAudit MCP/tool token weight across agents; warns on bloat (--agent, --json, --recommend, --profile-impact)
tokenix session-auditToken-economy health check: index, hook events, MCP/tool weight, cache hygiene
tokenix scan-secretsScan AI agent conversation transcripts for exposed credentials, gitleaks-style; attributes each to its repo + git branch (--agent, --filter, --group, --reveal, --json)
tokenix artifacts listList context artifacts defined in .tokenix/artifacts.json
tokenix artifacts show NAMEShow context artifact content
tokenix cyclesDetect circular dependencies in the symbol graph using Tarjan's SCC algorithm
tokenix rebuild-graphRebuild graph tables from existing chunks without re-embedding

⚙ Internal (invoked by hooks/agents, not by hand)

CommandDescription
tokenix hookPreToolUse handler — intercepts large reads, semantic grep, and noisy Bash commands (called by AI tools)
tokenix hook-postLegacy PostToolUse compatibility handler
tokenix run -- CMDRun a command and compress its output through tokenix filters
tokenix mcpMCP server exposing context, read/search, graph, and gain tools (--profile slim|full)

Global

| Flag | Description | |---|---|---| | --only-cpu | Force CPU embedding even on a GPU-enabled build (no-op on CPU-only builds) | | TOKENIX_BRANCH_AWARE=true | Env var: suffix SQLite DB per git branch (isolate indexes per branch) |

tokenix index — --force/-f, --cpu-profile <low\|default\|max>, --jobs N, --embed-batch N (default 16 CPU / 64 GPU), --if-stale, --path/-p, --model <id>, --no-low-priority (indexing runs at below-normal OS priority by default; this flag or TOKENIX_FOREGROUND=1 keeps normal priority)

Embedding model — default nomic-v1.5. Select another with tokenix index --model <id> or TOKENIX_EMBED_MODEL=<id>; run tokenix doctor to list available ids (nomic-v1.5, bge-small, bge-base, minilm-l6, e5-small, jina-code). The model is stamped into the index and read back at query time, so search always matches what was indexed; it is sticky across re-indexes and an explicit switch re-embeds. nomic-v1.5 (768d) is the quality default; bge-small (384d) indexes faster; e5-small is multilingual; jina-code is code-specialized (a custom ONNX downloaded from Hugging Face on first use). Existing indexes keep working unchanged.

tokenix query — --budget/-b (1200), --k (20), --file/-f, --link (cross-project, repeatable), --json, --path/-p

tokenix symbols — --limit/-l (20), --kind/-k <function\|struct\|class\|method\|...>, --json, --path/-p

tokenix deps — --reverse (files importing the target), --transitive (follow resolved imports), --json, --path/-p

tokenix grep — --limit/-l (20), --ignore-case/-i, --file/-f, --path/-p

tokenix context — --mode <plan\|debug\|audit\|security\|review>, --budget/-b (1200), --max-files, --budget-breakdown, --path/-p

tokenix impact — --depth/-d (2), --limit/-l (50), --format <text\|html\|mermaid\|json>, --output/-o, --path/-p

tokenix flow — --depth/-d (3), --format <text\|mermaid>, --output/-o, --path/-p

tokenix install-hook / remove-hook — --tool <claude-code\|copilot\|codex\|mcp\|antigravity\|all> (default all), --local (Claude Code, Copilot, and Antigravity)

tokenix pack — --mode/--profile <plan\|debug\|audit\|security\|review>, --budget N (8000), --format <markdown\|xml\|json>, --changed, --since REF, --token-map, --output/-o

tokenix benchmark — --budget N (1200), --json, --refresh-index, --cases FILE

tokenix prompt-audit — --agent <claude\|codex\|copilot\|antigravity\|all> (default all), --json, --recommend, --profile-impact

tokenix session-audit — --json, --cache-hygiene, --path/-p

tokenix scan-secrets — --agent <claude\|gemini\|copilot\|antigravity\|all> (default all), --filter <substr> (case-insensitive match over rule/agent/file/value/repo/branch), --group <none\|value\|rule\|agent\|file\|repo> (default none; value collapses each distinct secret into one block with its occurrence count, repo groups by the repository the secret was exposed in), --reveal (print raw values instead of redacting — warns on stderr), --json. Each finding is attributed to its repository + git branch when recoverable: Claude transcripts carry an exact cwd/gitBranch per message; otherwise the project directory is used as a best-effort ~slug:/~dir: label. Scans each agent's conversation transcripts under ~ (Claude ~/.claude/projects, Gemini ~/.gemini/tmp,history, Copilot ~/.copilot/session-state,logs, Antigravity ~/.gemini/antigravity) for credential patterns; output is redacted by default and exit code is 1 when findings exist. Patterns are TOML [[rules]] (id, pattern, optional capture/min_entropy): bundled defaults in assets/secret-rules/, extended/overridden by <repo>/.tokenix/secret-rules/*.toml then ~/.tokenix/secret-rules/*.toml (later sources win on matching id).

tokenix mcp — --profile <full\|slim> (default full)


🧠 Supported Languages

LanguageExtensionsSymbol types
Rust.rsfn, struct, enum, impl, trait, mod
Python.pydef, async def, class
TypeScript.ts, .tsxfunction, class, interface, type, arrow functions
JavaScript.js, .jsx, .mjs, .cjsfunction, class, arrow functions
Go.gofunc, type
C / C++.c, .cpp, .h, .hpp, .cc, .cxxfunction, class, struct, namespace
Config / Docs.toml, .md, .txt, .sh, .bashline blocks
Data files (opt-in).json, .yaml, .ymlIndexed only when data_files = true in .tokenix.toml
Customany extensionMapped to an existing parser via .tokenix.toml

Languages without a symbol-aware chunker (Java, C#, Ruby, Swift, Kotlin, …) are not indexed by default — blind line-block chunking produces low-quality search results.

Custom language mapping

Create a .tokenix.toml (or tokenix.toml) in the project root:

[languages]
pyi = "python"       # Python stub files
mts = "typescript"   # TypeScript module files
lua = "generic"      # use sliding-window chunks

Valid parser values: rust, python, typescript, javascript, go, cpp, c, generic.

Hook tuning

The same .tokenix.toml accepts a [hook] section to tune when the PreToolUse hook intercepts:

[hook]
read_min_lines = 120   # outline files with >= this many lines (default 200)
grep_min_words = 3     # treat Grep patterns with >= this many words as semantic (default 3)

Lower read_min_lines to intercept more reads (saving more tokens); raise it when you prefer verbatim file content. The fail-open contract is unchanged — hook errors never break the session.


🔧 Output Filters

tokenix reduces noisy shell output by rewriting matching Bash commands in PreToolUse so they run through tokenix run before the agent sees the result. Filtering happens in three layers (highest priority first):

  1. Local project filters — .toml files in .tokenix/filters/ inside the repo. Scoped to the project, committed to version control.
  2. User filters — .toml files in ~/.tokenix/filters/. Apply to all projects, override bundled filters.
  3. Bundled filters — 231 TOML output filters shipped inside the binary (each homologated against 500 embedded golden cases), covering uv, cargo build/cargo run, git, gradle, terraform plan, make, npm, pnpm, bun, deno, vite, node --test, poetry, docker, kubectl, helm, go, rust, python, dotnet, swift, and more. Applied automatically — no setup needed.

Filter format

[filters.uv-sync]
description = "Compact uv sync output"
match_command = "^uv\\s+(sync|pip\\s+install)\\b"
strip_ansi = true
strip_lines_matching = ["^\\s*$", "^\\s+Downloading ", "^\\s+Using cached "]
match_output = [
  { pattern = "Audited \\d+ package", message = "ok (up to date)" },
]
max_lines = 20
on_empty = "uv: ok"
FieldDescription
match_commandRust regex matched against the command. Compound commands are split (quote-aware) on &&, ||, ;, and |, and each segment is matched independently, so anchoring on the base command (e.g. ^gitleaks\b) still matches cd repo && gitleaks, cd repo;gitleaks, or producer | gitleaks
strip_ansiRemove ANSI colour codes before filtering
strip_lines_matchingDrop lines matching any of these regex patterns
keep_lines_matchingKeep only lines matching these patterns
match_outputShort-circuit: if output matches pattern, return message immediately
max_lines / head_lines / tail_linesTruncate output
truncate_lines_atTruncate individual lines at N characters
on_emptyMessage to return when filtering produces empty output
passthrough_when_emptiedWhen the filter reduces non-empty output to nothing (an unexpected output shape the keep/extract rules don't recognize), show a bounded view of the real output instead of on_empty — so format-specific filters never report a false "nothing here" (e.g. git log --oneline against the full-log filter)

AI-assisted filter generation

tokenix filter list                  # commands wasting the most tokens (no filter yet)
tokenix filter active                # all active user + bundled filters
tokenix filter record "cargo test"   # capture real output for richer generation
tokenix filter generate "cargo test" # generate a TOML filter via a local AI CLI

🏗 Architecture

src/
├── main.rs        CLI entry (clap), command dispatch, install-hook helpers
├── chunker.rs     Symbol-aware AST chunking (Tree-sitter) + dynamic language config (.tokenix.toml)
├── embed.rs       fastembed ONNX: embed_documents(), embed_query() — optional GPU via ort features
├── store.rs       SQLite schema, CRUD, FTS5, hybrid search, incremental branch fingerprint check
├── indexer.rs     File walker + incremental index pipeline (parallel chunking + batch embedding)
├── query.rs       Hybrid semantic + sparse FTS5 ranking, strict context modes, token-budget selection
├── pack.rs        Budgeted repo pack generation for non-hook AI tools, changed packs, token maps
├── graph.rs       Symbol relationship graph, cycle detection (Tarjan's SCC), HTML/Mermaid export
├── artifacts.rs   Context artifacts — parse `.tokenix/artifacts.json`, read non-code content
├── hook.rs        PreToolUse handler — Claude-, Copilot-, and grep_search/run_in_terminal-style JSON input
├── daemon.rs      Background TCP server — holds model + in-memory embedding cache
├── compress.rs    Legacy PostToolUse compatibility pipeline for tool-output rewriting
├── filters.rs     FilterDef, load_local/user/bundled_filters(), priority merge, apply_filter()
├── cmd_filter.rs  `tokenix filter` subcommands (list, active, generate, record)
├── recordings.rs  Capture/replay real command output for filter generation
├── memory.rs      Global/project preference memory (editable Markdown)
├── gain.rs        Analytics from the hook log — per-model cost table
├── benchmark.rs   Reproducible savings + retrieval-quality benchmark
├── doctor.rs      Backend / GPU / model-cache / daemon diagnostics
├── mcp.rs         Model Context Protocol server (full and slim profiles)
└── mcp_audit.rs   Multi-agent MCP config discovery + live tools/list introspection (prompt/session audit)

assets/
└── filters/       231 TOML output filters (+500 golden cases), embedded in the binary via rust-embed

GPU acceleration (opt-in)

A default build runs embeddings on CPU. Compile with a GPU feature to use the GPU — it then becomes the default at runtime, with automatic CPU fallback if the provider is unavailable:

# Windows — DirectML (works with any D3D12-capable GPU, no CUDA toolkit required)
cargo install --path . --features directml --locked

# Linux / Windows — CUDA (needs CUDA 12.x + cuDNN 9.x installed and on PATH;
# ort rc.9 does not support CUDA 13 yet)
cargo install --path . --features cuda --locked

On a GPU build, force CPU per-invocation with the global --only-cpu flag:

tokenix index .              # uses the GPU
tokenix --only-cpu index .   # forces CPU on a GPU build

--embed-batch drives peak memory (default 16 on CPU, 64 on GPU) — lower it if RAM/VRAM is tight. Run tokenix doctor to see the compiled backend, detected GPU, CUDA/cuDNN status, and tailored recommendations.

Daemon

The background daemon (tokenix serve) keeps the ONNX model and project embeddings in RAM (int8-quantized — 4x less memory than f32). Hook calls route over TCP loopback instead of re-loading the model on each subprocess invocation, and it auto-starts on the first Grep hook call — you don't need to run it manually. Manage it with tokenix daemon status (pid, port, uptime, model, cache size), tokenix daemon stop, and tokenix daemon restart.

Embedding model

PropertyValue
Modelnomic-embed-text-v1.5 (quantized)
Dimensions768
File size~130 MB
Cache location%LOCALAPPDATA%\tokenix\models (Windows) / ~/.cache/tokenix/models (Linux/macOS)
DownloadAutomatic on first run
Runtimefastembed (ONNX Runtime, in-process)

Index storage lives at ~/.tokenix/<project-id>.db (one DB per project). Embeddings are stored as int8-quantized blobs (4x smaller than f32, near-identical recall — the per-vector scale cancels out of the cosine) and similarity is computed in Rust — no external vector database needed. Indexes created before quantization migrate automatically (re-encode only, no re-embedding) on the next tokenix index; tokenix doctor reports migration coverage.


🔒 Security

tokenix's build and release pipeline is hardened against supply-chain attacks: SHA-pinned GitHub Actions, least-privilege workflow permissions, cargo-deny (advisories + license + crates.io-only sources), zizmor workflow analysis, OpenSSF Scorecard, SLSA build-provenance attestations, and tokenless crates.io publishing via OIDC. See SECURITY.md for the disclosure policy and release-verification steps.


🤝 Contributing

Contributions are welcome! See CONTRIBUTING.md for how to get started.


📄 License

MIT

Ver o original no GitHub

Bom para

Relacionados