convez.ai
MCP ServerDados & AnalyticsAvançado
blackwell-systems

BetterThanTOON - Benchmark GCF vs TOON

blackwell-systems/betterthantoon

Benchmark técnico: GCF vence TOON em compreensão, geração e tokens

Ver no GitHub
InicialPluginScore · de 0 a 100

Repositório com dados de benchmark completos comparando GCF e TOON em compressão de dados tabulares. Inclui métricas de compreensão, geração e eficiência de tokens em múltiplos modelos (Claude, GPT, Gemini), demonstrando superioridade técnica do GCF em leitura, escrita e compactação de dados.

Instalação

$npx convez-ai blackwell-systems/betterthantoon

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
28%
Último commit
há 4 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Comparar formatos de serialização de dados tabulares para LLMs
  • Avaliar eficiência de tokens em diferentes estruturas de dados
  • Validar confiabilidade de formato em geração de modelos
  • Fundamentar decisões técnicas sobre compressão de dados

Veredito

Análise técnica rigorosa com dados reproduzíveis mostrando falhas sistêmicas do TOON em geração (0/5 em múltiplos modelos) e compreensão. GCF demonstra vantagem consistente, mas o valor prático depende de adequação ao caso específico de uso — não é conclusivo para todos os cenários tabulares.

README

A tradução para o português deste README ainda está na fila.

betterthantoon.com

GCF is better than TOON on every dimension. Their datasets, their tokenizer, their methodology. Full benchmark data proving it.

Why TOON fails

TOON claims ~40% token savings over JSON on tabular data. What they don't measure: whether LLMs can actually read it at scale, or produce valid output in it.

  • Comprehension: TOON averages 68.5% at 500 records. GCF averages 90.7%.
  • Generation: TOON's own decoder rejects LLM-generated output on 7 of 9 models. The error is always the same: toon: cannot assign string to int. The model writes "target" where TOON expects the integer 0.
  • Token efficiency: GCF wins all 6 datasets on TOON's own benchmark.

TOON's flat tabular design forces two failure modes that GCF eliminates structurally:

  1. Distance grouping failure: without section headers, models must scan 500 rows and filter by column value. They guess round numbers instead. TOON median error: 53.
  2. Semantic-to-integer encoding: TOON encodes categories as integers with no structural cue. Models can't perform this mapping unprompted.

Token efficiency (TOON's own benchmark)

Using TOON's benchmark code, TOON's tokenizer (o200k_base), and TOON's datasets:

DatasetGCFTOONResult
Semi-uniform event logs108,158154,032GCF 42% smaller
E-commerce orders61,59373,246GCF 19% smaller
Deeply nested config616618GCF 0.3% smaller
Employee records49,05549,966GCF 2% smaller
Analytics time-series8,3989,127GCF 8% smaller
GitHub repos8,5768,744GCF 2% smaller

GCF wins all 6. Including the flat tabular datasets TOON was designed for.

Comprehension (can the model read it?)

500 symbols, 200 edges, 13 extraction questions. No format instructions. No system prompt.

ModelGCFTOONJSON
Claude Opus 4.696.2%84.6%73.1%
Claude Sonnet 4.6100%73.1%53.8%
Claude Haiku 4.596.2%69.2%57.7%
GPT-5.584.1%67.7%45.8%
GPT-5.478.0%56.0%44.1%
GPT-5.4-mini71.8%64.1%54.2%
Gemini 2.5 Pro100%76.9%58.3%
Gemini 3.1 Pro100%76.9%46.2%
Gemini 3.5 Flash100%61.5%46.2%
Gemini 2.5 Flash80.6%54.6%57.0%

23 runs, 10 models, 3 providers. GCF wins 22, ties 1, loses 0. Four models hit 100%.

Generation (can the model write it?)

3-line primer, validated through real decoders. Same data, same prompt structure.

ModelGCFTOONJSON
Claude Opus 4.65/50/55/5
Claude Sonnet 4.65/52-3/55/5
Claude Haiku 4.55/51-3/55/5
GPT-5.54-5/51-2/55/5
GPT-5.45/50/55/5
GPT-5.4-mini5/50/55/5
Gemini 2.5 Pro5/51/55/5
Gemini 3.1 Pro5/50/55/5

No model has ever been trained on GCF. Every frontier model produces valid output on first exposure. TOON has been published for months and fails on 7 of 9 models.

The TOON generation failure

Every TOON generation failure produces the same error:

INVALID: symbols: index 0: distance: toon: cannot assign string to int

The model writes target in the distance column. TOON expects 0. The model would need to know, unprompted, that "target" maps to 0, "related" maps to 1, "extended" maps to 2. No model does this because the format gives no structural cue.

GCF expresses distance through section placement: targets go in ## targets, related in ## related. No integer mapping required.

Even when TOON is hand-held (pre-encoded integers in the prompt), GCF output is still 28% smaller.

What GCF has that TOON doesn't

  • Session deduplication: symbols from prior calls become bare references. 92.7% savings by the 5th call. TOON re-serializes everything every time.
  • Delta encoding: 81.2% additional savings on re-queries. TOON has no equivalent.
  • Hierarchical grouping: section headers eliminate the distance-filtering failures that plague TOON at scale.
  • Local IDs: edges cost ~4 tokens each regardless of identifier length. TOON repeats full qualified names.
  • Streaming: zero-buffering with trailer summary. TOON has no streaming support.

We opened a PR on TOON's repo

We added GCF as a formatter to TOON's official benchmark. 9 lines of code. The PR is public:

toon-format/toon#319

The fork with full results:

blackwell-systems/toon

How to try GCF

pip install gcf-proxy

Wrap any MCP server. Zero code changes. Or use the libraries directly:

pip install gcf-python        # Python
npm install @blackwell-systems/gcf  # TypeScript
go get github.com/blackwell-systems/gcf-go  # Go
cargo add gcf                 # Rust

Links

License

MIT - Dayna Blackwell

Ver o original no GitHub

Bom para

Relacionados