LexonArchiveBuilder
Alan-Jowett/LexonArchiveBuilder
Indexa e busca em arquivos de email e documentos via grafo semântico
Ver no GitHubMCP server que estrutura arquivos de email, RFCs e documentos em um grafo de conhecimento semântico unificado e consultável através de LexonGraph. Funciona como camada de aplicação para indexação e busca semântica em coleções de discussões e documentos técnicos.
Instalação
npx convez-ai Alan-Jowett/LexonArchiveBuilder1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Indexação semântica de arquivos de email e listas de discussão
- Busca e recuperação em coleções de documentos técnicos (RFCs, specs)
- Construção de camada de conhecimento para workflows RAG
- Pesquisa semântica em threads e mensagens estruturadas
Veredito

Projeto sólido e bem arquitetado para indexação e busca semântica em arquivos estruturados. Ideal para quem precisa transformar coleções fragmentadas de emails e documentos em uma camada de conhecimento consultável; exige familiaridade com Rust, Docker e conceitos de grafos semânticos.
README
A tradução para o português deste README ainda está na fila.
LexonArchiveBuilder
LexonArchiveBuilder weaves mail archives, RFCs, and metadata into a unified, queryable knowledge layer atop LexonGraph, structuring threads, messages, and chunks into a coherent semantic fabric accessible through an MCP server.
LexonArchiveBuilder is the application layer built on top of LexonGraph. It exists to prove that LexonGraph works on real data and to provide a practical system for indexing and semantically searching mail archives and technical document collections.
At a high level, LexonArchiveBuilder has two jobs:
- Index content into a structured semantic graph.
- Serve search and retrieval through an MCP server.
The initial content types are email and documents, but the architecture is intended to extend to additional content types over time without changing the core search contract.
Why it exists
LexonArchiveBuilder is meant to validate LexonGraph under realistic ingestion and retrieval workloads while also being directly useful as an application in its own right. The project focuses on turning fragmented archives, messages, and documents into a coherent knowledge layer that can support semantic lookup, retrieval, and downstream RAG-style workflows.
Architecture overview
LexonArchiveBuilder is designed as a CDN-backed RAG system with:
- LexonGraph as the underlying graph and knowledge substrate
- An indexer that ingests source content, extracts structure, and emits searchable graph-aligned artifacts
- An MCP server that exposes search and retrieval over that indexed knowledge
- Environment-specific adapters for storage and embeddings
The system is intentionally split so that indexing remains separate from search serving. Outside of indexing, the intended shape is to avoid a central control plane or other server-side processing layers.
Content model
The current focus is on content that naturally decomposes into:
- collections
- threads
- messages
- documents
- chunks
- metadata
That model supports archives of discussion content alongside long-form technical documents while keeping the design open to future content types.
Environment model
| Environment | Storage | Embeddings | Runtime shape |
|---|---|---|---|
| Local / testing | Local filesystem | Local embedding server such as ghcr.io/substratusai/stapi | 100% local, with indexing running inside Linux Docker containers |
| Production | Azure Blob Storage | Azure OpenAI | Planned/TBD batch-oriented Azure container application shape |
This split is intentional: local development should be self-contained and easy to run without cloud dependencies, while production should scale against cloud-native storage and embedding services.
MCP server
The MCP server is the search-facing surface of LexonArchiveBuilder. It is intended to:
- expose the indexed knowledge layer through a stable MCP interface
- remain compatible with both Linux and Windows
- support content backed by either the local filesystem or Azure Blob Storage
The goal is for clients to interact with a consistent semantic search surface even as the underlying storage and embedding providers vary by environment.
Local development and testing
Local and testing workflows are designed to be fully local:
- source content lives on the local filesystem
- indexing runs inside Linux Docker containers
- embeddings come from a local embedding service, currently expected to be something like
ghcr.io/substratusai/stapi - the MCP server should remain usable from Linux or Windows environments
This keeps local development fast, reproducible, and independent of cloud services.
Production direction
Production is intended to use:
- Azure Blob Storage for persisted content and artifacts
- Azure OpenAI for embeddings
- a batch-oriented Azure container app deployment model
Some production details are still planned/TBD, especially the exact batch shape and surrounding operational workflow. This README describes the intended direction without claiming those deployment details are finalized.
Using LexonArchiveBuilder
Conceptually, LexonArchiveBuilder is used in two stages:
- Indexing: ingest mail archives and technical document collections, normalize them into the LexonGraph-backed fabric, and generate searchable semantic artifacts.
- Querying: connect through the MCP server to search threads, messages, documents, chunks, and related metadata through a unified knowledge layer.
Concrete commands and operational examples can be added as the executable surface of the repository stabilizes.
Indexer MVP
The first lexonarchivebuilder-indexer MVP is now implemented as a Rust batch runtime in
crates/lexonarchivebuilder-indexer.
Request contract
The MVP request format is JSON and remains collection-oriented across both supported content classes:
- mailbox items point at per-month
.mboxor.mailfiles - document items point at
.txtfiles
See examples/local/request.sample.json for a complete local request that
indexes one mailbox file and one document file.
Running locally
Build and run the batch directly:
cargo run -p lexonarchivebuilder-indexer -- run --request examples\local\request.sample.json
The sample request assumes the embedding endpoint is available at
http://stapi:8080. For manual testing against an already-running STAPI
container, update the request file's base_url to match that endpoint.
Running with Docker Compose
The repository includes a local integration stack:
docker compose up --build indexer
That stack starts:
stapiathttp://localhost:8080- the
lexonarchivebuilder-indexerbatch container - a named Docker volume mounted into the batch container at
examples/local/block-store
After the batch completes, the summary output is written to
examples/local/output/summary.json.
Local scale-test workflow
For large local mailbox stress tests, the repository also includes a lightweight
wrapper script that fetches one or more rsync-backed mailbox archives, discovers
.mail and .mbox files, generates an indexer request, and runs the existing
lexonarchivebuilder-indexer batch to produce a block tree and summary/root handoff
artifact.
The wrapper is intentionally simple. It is designed as a local stress-test harness over the existing indexer rather than a new indexer subsystem.
Prerequisites
- Direct Linux script entrypoint:
rsyncon the host plus Docker withdocker compose - Docker Compose entrypoint on Linux or Windows: Docker with
docker compose
Run it with one or more rsync URLs:
scripts/lexonarchivebuilder-scale-test.sh \
rsync.ietf.org::mailman-archive/ipsec/
Or from a sources file:
scripts/lexonarchivebuilder-scale-test.sh \
--sources-file examples/local/scale-test/rsync.sources.sample.txt
The wrapper also exposes the existing indexer clustering flags as first-class script arguments. For example:
scripts/lexonarchivebuilder-scale-test.sh \
--clustering-mode divisive \
--clustering-algorithm directional-pca \
--clustering-cluster-count 3 \
--clustering-random-seed 7 \
--clustering-retained-dimension-count 1 \
--clustering-variance-exponent 1.5 \
--clustering-temperature 0.75 \
--clustering-min-input-count 2 \
--clustering-min-effective-rank 1 \
--clustering-min-cumulative-variance 0.25 \
rsync.ietf.org::mailman-archive/ipsec/
On Linux, you can also launch the same workflow through Docker Compose:
docker compose run --rm scale-test
On Windows, use the Docker Compose entrypoint:
docker compose run --rm scale-test
To point the Compose entrypoint at a different sources file in the repository:
docker compose run --rm scale-test --sources-file /workspace/examples/local/scale-test/rsync.sources.sample.txt
The same clustering flags can be passed through the Docker Compose entrypoint:
docker compose run --rm scale-test `
--sources-file /workspace/examples/local/scale-test/rsync.sources.sample.txt `
--clustering-mode aggregation `
--clustering-algorithm dcbc `
--clustering-cluster-count 2 `
--clustering-random-seed 11
Each run writes its generated request, fetched mailbox mirror, block store, and summary output under:
examples/local/scale-test/runs/<run-id>/
The generated request.json is compatible with the existing local indexer
contract, and summary.json contains the resulting root_id for the produced
block tree.
To exercise the wrapper end to end against local mailbox fixtures, run:
scripts/lexonarchivebuilder-scale-test-smoke.sh
To exercise the Docker Compose entrypoint end to end against local mailbox fixtures, run:
scripts/lexonarchivebuilder-scale-test-compose-smoke.sh
MCP MVP
The first lexonarchivebuilder-mcp MVP is now implemented as a Rust stdio MCP server
in crates/lexonarchivebuilder-mcp.
Request contract
The local MVP server reads a JSON config file that identifies:
- the local filesystem-backed block store to search
- the local OpenAI-compatible embedding endpoint to use for query embeddings
- the index summary file that provides the current
root_id - default
top_kand traversal-width settings for chunk search
See examples/local/mcp.request.sample.json for a complete local config.
Running locally
First generate the local block store and summary:
cargo run -p lexonarchivebuilder-indexer -- run --request examples\local\request.sample.json --summary-out examples\local\output\summary.json
The sample MCP config uses http://stapi:8080 for the Docker Compose network.
For a host-side cargo run, update examples\local\mcp.request.sample.json to
use a host-reachable endpoint such as http://localhost:8080 before starting
the server.
Then start the MCP server over stdio:
cargo run -p lexonarchivebuilder-mcp -- serve --config examples\local\mcp.request.sample.json
The MVP exposes four MCP tools:
search_chunksget_documentget_emailget_thread
The search tool is executable end to end in the local profile. The named
retrieval tools are present in the MVP surface and currently return an explicit
unsupported outcome until LexonGraph exposes a delegated retrieval-by-name
contract for those item classes.
Running with Docker Compose
Use the local integration stack in three steps so STAPI stays available in the
background, the one-shot indexer finishes writing
examples/local/output/summary.json, and then the MCP server starts:
docker compose up -d stapi
docker compose run --rm --build --no-deps indexer
docker compose run --rm -i --no-deps mcp
That workflow uses:
stapipublished to the host athttp://localhost:8080and reached from the Compose network ashttp://stapi:8080- the
lexonarchivebuilder-indexerbatch container - the
lexonarchivebuilder-mcpstdio server container - a named Docker volume mounted into both containers at
examples/local/block-store
License
This project is licensed under the MIT License. See LICENSE.