convez.ai
MCP Server
AliceLJY

babel-memory

AliceLJY/babel-memory

Ver no GitHub
InicialPluginScore · de 0 a 100

Fix the multilingual blind spot in AI memory systems. BM25 pre-tokenization for CJK (jieba/kuromoji), Snowball stemming for 20 European languages, bilingual KG/session prompts. Zero required deps.

Instalação

$npx convez-ai AliceLJY/babel-memory

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 3 semanas
Versão
v2.1.2
Issues abertas
0
Contribuidores
1

README

A tradução para o português deste README ainda está na fila.

babel-memory

The first standalone library fixing the multilingual blind spot in AI memory systems.

27+ languages. Zero required dependencies. Drop-in fix for BM25 + RAG.

Not affiliated with Babel.js. Named after the Tower of Babel — breaking the language barrier in AI agent memory.

npm npm downloads License: MIT GitHub stars Languages Dependencies

English | 简体中文


Why This Exists

Every major AI memory / RAG system today — mem0, Letta, LanceDB-based stores — silently fails on non-English content. Research across 8 academic papers (MMTEB, XRAG, MIT 2025) reveals a systematic 5-layer semantic loss cascade:

LayerWhat BreaksImpact
Token estimationstring.length / 4 underestimates CJK by 4-8xContext overflow
BM25 tokenizationWhitespace split on Chinese = 0 matchesHybrid search degrades to vector-only
LLM extractionEnglish-only KG/summary prompts-24% factual accuracy on non-English
Cross-lingual retrievalQuery/document language mismatch-56% recall (XRAG benchmark)
Auto-evaluationLLM-as-Judge overestimates non-English qualityProblems go systematically unreported

babel-memory is the fix for layers 2-4. Same simple API. Zero required dependencies. Install only the language packs you need.

Before & After

BEFORE babel-memory:
  Store: "机器学习在自然语言处理中的应用"
  BM25 search("机器学习") → [] (zero results)
  KG extract → English prompt struggles with Chinese entities
  
AFTER babel-memory:
  Store: "机器学习在自然语言处理中的应用"
         → fts_text: "机器 学习 机器学习 自然 语言 处理 自然语言 应用"
  BM25 search("机器学习") → [match found!]
  KG extract → Chinese prompt with CJK few-shot examples
BEFORE babel-memory (European):
  Store: "Maschinelles Lernen verbessert die Verarbeitung"
  BM25 search("Verarbeitung") → [match]
  BM25 search("verarbeitet") → [] (different form, zero results)

AFTER babel-memory + snowball-stemmers:
  Store: "Maschinelles Lernen verbessert die Verarbeitung"
         → fts_text: "maschinell lern verbess verarbeit"
  BM25 search("verarbeitet") → stem("verarbeitet") = "verarbeit" → [match found!]

Modular Install

# Core only (zero dependencies — pure TypeScript)
npm install babel-memory

# Add language packs as needed:
npm install jieba-wasm          # Chinese (highest quality)
npm install @sglkc/kuromoji     # Japanese (highest quality)
npm install wordcut             # Thai
npm install snowball-stemmers   # 20 European languages (German, French, Spanish, Russian, etc.)
npm install tinyld              # auto-detect Latin-script languages (de/fr/es/...)

You only pay for what you use. The core package has zero dependencies — language packs are loaded lazily at runtime. And thanks to the built-in Intl.Segmenter tier (ICU dictionaries shipped with Node 16+/Bun/Deno/browsers), a zero-dependency install already gets word-level segmentation for Chinese, Japanese and Thai — installing the packs upgrades quality further. It never crashes.

Quick Start

import { detectLanguage, initTokenizer, tokenizeForFts, getKgPrompt } from "babel-memory";

// 1. Initialize once at startup (loads whichever packages are installed)
await initTokenizer();

// 2. Detect language (zero dependencies, pure Unicode analysis)
detectLanguage("这个项目的架构设计非常优秀");  // "zh"
detectLanguage("東京タワーはとても高いです");    // "ja" (not "zh" — hiragana detected first)
detectLanguage("이 프로젝트는 매우 훌륭합니다"); // "ko"
detectLanguage("สวัสดีครับ");                    // "th"
detectLanguage("مرحبا بالعالم");                 // "ar"
detectLanguage("Машинное обучение");              // "ru"

// 3. Pre-tokenize for BM25 (the core fix)
tokenizeForFts("机器学习很有趣", "zh");
// → "机器 学习 很 有趣"  (jieba word segmentation)

tokenizeForFts("東京タワー", "ja");
// → "東京 タワー"  (kuromoji word segmentation)

tokenizeForFts("Maschinelles Lernen", "de");
// → "maschinell lern"  (Snowball stemming)

// 4. ⚠️ CRITICAL: apply the SAME tokenization to your queries.
// Tokenization only matches when index side and query side agree:
const ftsQuery = tokenizeForFts(userQuery, detectLanguage(userQuery));
// store side:  "机器学习很有趣" → "机器 学习 很 有趣"  (indexed)
// query side:  "机器学习"       → "机器 学习"          (matches!)
// forgetting this = querying one giant token "机器学习" against
// segmented text = zero results. This is THE classic FTS pre-tokenization trap.

// 5. Get language-matched prompts for LLM calls
const { system, userTemplate } = getKgPrompt("zh"); // → Chinese prompt
getKgPrompt("ja"); // → native Japanese prompt
getKgPrompt("ko"); // → English prompt (instructions in a third language hurt quality)
// Predicates stay English (normalized keys) in every variant

// 6. Optional power tools
import { detectLanguageDetailed, detectLanguageExtended, getLoadedTokenizers } from "babel-memory";

detectLanguageDetailed("我在用 TypeScript 给 RecallNest 写 tokenizer hook");
// → { language: "en", scripts: { cjk: 0.13, latin: 0.83, ... }, isMixed: true }
// AI conversation logs mix CN/EN constantly — isMixed tells you when.
// Segmentation-needing scripts (CJK/Thai) flag mixed from a ~2% ratio:
// even a tiny Chinese island is unsearchable if left unsegmented.
// (Embedded CJK runs are segmented automatically either way; see below.)

detectLanguageExtended("Das ist ein guter Tag");
// → "de" when tinyld is installed; "en" otherwise (graceful degrade)

tokenizeForFts("机器学习的应用", "zh", { removeStopwords: true });
// → "机器 学习 应用"  (的 removed; opt-in, default off)

getLoadedTokenizers(); // → ["jieba", "kuromoji", "tinyld"] — verify deployment

// Selective init when you don't need everything (kuromoji alone = ~1-2s + ~17MB):
await initTokenizer({ languages: ["zh", "de"] });

How It Works

The key insight: pre-tokenize non-whitespace-delimited text before FTS indexing, and stem inflected languages.

Standard FTS pipeline (broken for Chinese):
  "知识图谱提取" → whitespace split → ["知识图谱提取"] → 1 giant token → no matches

babel-memory pipeline (fixed):
  "知识图谱提取" → jieba segmentation → "知识 图谱 知识图谱 提取" → whitespace split → 4 tokens → matches!

This works with any whitespace-based FTS engine: Tantivy (LanceDB), SQLite FTS5, Elasticsearch, Meilisearch. No engine modifications needed.

Measured: the gap is not subtle

bun bench/recall-benchmark.ts — SQLite FTS5, 20 Chinese documents, 12 queries:

TierRecallQueries returning nothing
raw text (what most memory systems do)0.0%12/12
zero-dependency tier (Intl.Segmenter + bigrams)100%0/12
full tier (jieba)100%0/12

Raw Chinese text in a whitespace FTS engine doesn't degrade — it fails completely. Every single query returns empty.

Mixed-script text (the AI-conversation reality)

Real agent conversations constantly mix languages: "I fixed 机器学习模型 using TensorFlow". Ratio-based detection lands on zh or en depending on the balance — and either route now handles the mix correctly: the zh path's tokenizers keep Latin tokens intact, and the en path detects embedded CJK/Hangul/Thai runs and routes each run to its proper tokenizer. Before v2.1, a Chinese island inside en-classified text passed through unsegmented and unsearchable. Use detectLanguageDetailed() if you want the mixing signal explicitly.

Detection Order Matters

Japanese uses kanji (CJK characters). Naive CJK detection would misclassify Japanese as Chinese. babel-memory checks language-unique scripts first:

  1. Hiragana/Katakana present? → Japanese (unique to Japanese)
  2. Hangul present? → Korean (unique to Korean)
  3. Thai script? → Thai
  4. Arabic script? → Arabic
  5. Devanagari? → Hindi
  6. Cyrillic? → Russian
  7. CJK Ideographs without Japanese/Korean markers? → Chinese
  8. Default → English

Graceful Degradation

babel-memory never crashes due to a missing optional package. Each language has a three-tier fallback chain — and since v2.1 the middle tier (ICU via Intl.Segmenter, built into Node 16+/Bun/Deno) gives zero-dependency installs word-level quality:

LanguageTier 1: package installedTier 2: zero deps (built-in ICU)Tier 3: last resort
Chinesejieba search-mode segmentationICU word segmentation + CJK bigramsCharacter split
Japanesekuromoji segmentationICU word segmentation + CJK bigramsCharacter split
Thaiwordcut segmentationICU word segmentationPassthrough
European (de, fr, es...)Snowball stemmingPassthroughPassthrough
Korean—Syllable-level split (deliberate: see note)Same
Arabic, RussianSnowball stemmingPassthroughPassthrough
Hindi—Passthrough (space-delimited)Same
English—PassthroughSame

Why bigrams? ICU emits whole compounds ("東京タワー" as one token), which breaks partial-match queries like "タワー". CJK tokens of length ≥ 3 get bigram expansion on both index and query side — the same approach lunr-languages uses.

Why is Korean syllable-level? Korean is agglutinative: "프로젝트는" = "프로젝트" + topic particle. Word-level tokens make the query "프로젝트" miss entirely; syllable split keeps BM25 partial matching stable.

A warning is logged once per missing package so you know what to install for better quality. Your application keeps working regardless.

API Reference

FunctionInputOutputDescription
detectLanguage(text)stringLanguageUnicode script ratio analysis. Detects zh, ja, ko, th, ar, hi, ru, en. Zero dependencies.
detectLanguageDetailed(text)stringLanguageDetailSame + per-script ratios and isMixed flag for mixed-script text.
detectLanguageExtended(text)stringstringRefines Latin-script text to de/fr/es/... when tinyld is installed; otherwise identical to detectLanguage.
initTokenizer(opts?){ languages?: string[] }Promise<void>Load available tokenizers in parallel. Pass languages to load selectively. Idempotent, non-fatal.
tokenizeForFts(text, lang, opts?)string, string, { removeStopwords? }stringNFKC-normalize, then pre-tokenize for BM25. Apply to both indexed text and queries.
getLoadedTokenizers()—string[]Diagnostic: which optional packs are actually loaded.
segmentWithIntl(text, locale)string, stringstring | nullRaw ICU word segmentation building block. null when unavailable.
intlWithBigrams(text, locale)string, stringstring | nullICU segmentation + CJK bigram expansion (what the zh/ja fallback uses).
getKgPrompt(lang)string{ system, userTemplate }KG triple extraction prompt. zh → Chinese, ja → Japanese, others → English.
getSessionPrompt(lang)string{ system, dimensionLabels }Session summary prompt, same language routing. 9 structured dimensions.

Type: Language = "zh" | "ja" | "ko" | "th" | "ar" | "hi" | "ru" | "en"

tokenizeForFts also accepts any Snowball language code (e.g., "de", "fr", "es") as a string.

Note on script-based detection limits: detectLanguage cannot distinguish languages that share the Latin alphabet — German, French and Spanish all return "en". That's why detectLanguageExtended + the optional tinyld pack exist: install it and Latin-script languages resolve automatically, completing the auto-detect → Snowball stemming chain. Traditional Chinese note: ICU and jieba both handle zh-Hant text, but jieba's dictionary is Simplified-centric; for mixed Simplified/Traditional corpora consider normalizing externally (OpenCC) before indexing.

Supported Languages

Auto-Detected (via detectLanguage)

CodeLanguageScriptFTS StrategyPackage
zhChineseCJK Ideographsjieba search-mode word segmentationjieba-wasm
jaJapaneseHiragana + Katakana + CJKkuromoji word segmentation@sglkc/kuromoji
koKoreanHangul + CJKCharacter-level split(built-in)
thThaiThai scriptwordcut segmentationwordcut
arArabicArabic scriptSnowball stemmingsnowball-stemmers
hiHindiDevanagariPassthrough(none)
ruRussianCyrillicSnowball stemmingsnowball-stemmers
enEnglishLatinPassthrough(none)

Snowball-Stemmed Languages (pass code to tokenizeForFts)

CodeLanguageCodeLanguage
deGermannlDutch
frFrenchsvSwedish
esSpanishnoNorwegian
ptPortuguesedaDanish
itItalianfiFinnish
huHungariantrTurkish
roRomaniancsCzech

Total: 8 auto-detected + 14 explicit Snowball = 27+ languages (Arabic and Russian appear in both lists).

With the optional tinyld pack installed, detectLanguageExtended() auto-detects the Latin-script languages too — closing the loop so all 27+ become reachable without the caller knowing the language up front.

Who Is This For

  • AI memory system builders — if you're building on LanceDB, ChromaDB, or any vector+BM25 hybrid store
  • RAG pipeline developers — if your users speak non-English languages and BM25 returns empty
  • MCP server authors — if your memory tools need multilingual support
  • Anyone who's noticed their AI agent "forgets" non-English conversations

Compared to Alternatives

Most AI memory / RAG systems treat tokenization as solved. They're not wrong — for English. For the rest of the world:

babel-memorymem0LettaRaw LanceDB FTS
CJK word segmentationjieba / kuromoji, ICU built-in fallbackNoneNoneCharacter bigrams
Zero-install CJK qualityWord-level (Intl.Segmenter + bigrams)——Character bigrams
Mixed CN/EN textScript-run routingNoneNoneNone
European stemmingSnowball (20 langs)NoneNoneNone
Language detection8 scripts + optional tinyld (62 langs)NoneNoneNone
Language-matched LLM promptsEN + ZH + JAEnglish onlyEnglish onlyN/A
Required dependencies0HeavyHeavyN/A
Works with any FTS engineTantivy, SQLite FTS5, ES, MeilisearchLocked inLocked inLanceDB only
Measured Chinese BM25 recall100% (vs 0% raw — see bench/)unmeasuredunmeasuredpartial

babel-memory is not a memory system — it's a preprocessing layer that makes any memory system work properly across languages.

Used By

  • RecallNest — MCP-native shared memory for Claude Code, Codex, and Gemini CLI

Research References

This library is informed by findings from:

  • MMTEB: Massive Multilingual Text Embedding Benchmark (arXiv 2502.13595)
  • XRAG: Cross-lingual Retrieval-Augmented Generation (arXiv 2505.10089)
  • MIT: Tokenization Changes Meaning in LLMs (Computational Linguistics, 2025)

License

MIT

Ver o original no GitHub

Bom para

Relacionados