convez.ai
MCP Server
unison-labs-ai

Unison-evals

unison-labs-ai/Unison-evals

Ver no GitHub
InicialPluginScore · de 0 a 100

Open benchmark suite for AI memory and agent systems — reproducible methodology, results reported in full.

Instalação

$npx convez-ai unison-labs-ai/Unison-evals

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
50%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

README

A tradução para o português deste README ainda está na fila.

unison-evals

CI License Python 3.12

Public benchmark harness for Unison and comparable agent / memory systems.

Treats the production agent as a black box: every system implements one adapter (~80 LOC), points at its API/CLI, and is scored on the same datasets with the same metrics under the same constraints.

v0.1 — early but real. Ships 4 datasets (LongMemEval, LOCOMO, MemoryAgentBench, Context-Bench) across 2 tracks (agent-oracle, agent-e2e), with adapters for unison-agent and unison-agent-pipeline. The hosted leaderboard is next.

Repository structure

Two parallel eval systems live here, intentionally separate:

SubpackageWhat it evaluatesStyle
src/unison_evals/memory_evals/Single-question Q&A over a per-question corpusTrack 2 (oracle) + Track 3 (E2E)
src/unison_evals/benchmarks/Task-shaped, multi-turn, end-state-scored benchmarksτ-bench, Letta Context-Bench, etc.

The two share infrastructure (cli.py, config.py, types.py, server/) but have different adapter contracts and scoring shapes. Pick the one that matches the question you're asking:

  • "Does my agent retrieve and reason well over a corpus given a question?" → memory_evals/
  • "Does my agent's interface let it accomplish a multi-turn task correctly?" → benchmarks/

See each subpackage's README for details:

Top-level docs reference

FileWhat it is
README.md (this file)quickstart + repo structure
METHODOLOGY.mdhow scores are computed, hardware, datasets

What it measures

Two tracks isolate two different failure modes:

TrackWhat it testsMethod
agent-oracleReasoning given perfect contextHand the agent gold context → score answer (no retrieval)
agent-e2eWhat users actually experienceAgent ingests per-question corpus, retrieves, answers → score answer + cost + latency

What you're comparing in agent-e2e

agent-e2e = "given the same per-question corpus, does the agent retrieve and reason correctly?"

Every system receives the identical per-question document corpus via seed_docs. The agent ingests them, retrieves relevant content, and answers — scored by the LLM judge.

AdapterWhat it does
unison-agentIngests docs into a brain via seedDocs; retrieves only what's relevant for the question
unison-agent-pipelineSame agent, different ingestion pipeline configuration
unison-brain-contextNew brain-only contract (post restructure): provision→seed via /v1/eval/seed→GET /v1/brain/context→reader LLM answers

The headline metric is pass_rate — the fraction of questions the agent answered correctly per the LLM judge.

Results

These are end-to-end answer accuracy numbers (LLM-judge): the agent ingests the corpus, retrieves, and answers, and the judge grades the final answer against ground truth. That's the hard metric — not retrieval recall@k (whether the right snippet was fetched). A recall@k number is not comparable to answer accuracy: you can retrieve the right passage and still answer wrong. Judge: gemini-3.1-flash-lite (the --dev judge); run-to-run variance ≈ ±2–3pp.

LOCOMO

Dataset: snap-research/locomo — original locomo10.json (paper, ACL 2024). Scored on categories 1–4 (single-hop, multi-hop, temporal, open-domain); adversarial (cat 5) excluded — it is ungradeable (444/446 have no ground-truth answer). The full Unison agent ingests each conversation once, retrieves, and answers. n=128, proportional, seed 1234.

MetricAnswer accuracy
Overall (cats 1–4, n=128)85.9%
open-domain93% (n=70)
multi-hop81% (n=27)
single-hop78% (n=23)
temporal62% (n=8)

LOCOMO carries ~6.4% documented label errors that affect any system scored on it (penfieldlabs audit).

EVAL_STRATIFIED=proportional EVAL_SEED=1234 \
  uv run unison-evals run --dataset locomo --systems unison-agent --limit 128 --dev

LongMemEval

Dataset: LongMemEval (ICLR 2025), longmemeval_s_cleaned — the hard split with full ~50-session haystacks and distractors. The full Unison agent ingests → retrieves → answers. n=150, proportional, seed 9012.

MetricAnswer accuracy
Overall (n=150)91.3%

Reproducing unison-agent. The harness is open source, but unison-agent runs against a Unison brain server that is authenticated and not publicly hosted — request an eval access token by emailing misha@unisonlabs.ai (briefly state your use case), then point the harness at the provided server:

export UNISON_API_URL=...        # provided with your token
export UNISON_EVAL_SECRET=...    # your eval access token
EVAL_STRATIFIED=proportional EVAL_SEED=9012 \
  uv run unison-evals run --dataset longmemeval --systems unison-agent --limit 150 --dev

Additional comparator adapters can be added via the adapter interface — see Adding a new system below.

Quickstart

# 1. Clone + install
git clone https://github.com/unison-labs-ai/Unison-evals.git
cd Unison-evals
uv sync                                   # uses .python-version (3.12)

# 2. Configure
cp .env.example .env
$EDITOR .env                              # set UNISON_JWT, ANTHROPIC_API_KEY
# To run the `unison-agent` system, you need a Unison brain server — get the
# open-source server + clients at https://github.com/unison-labs-ai/unison-brain
# Tip: when running against a local Unison server, leave UNISON_JWT blank and set
# UNISON_LOCAL_EVAL_TENANT_ID in the brain server's .env — no JWT needed.

# 3. Run an eval — CLI
uv run unison-evals run \
  --systems unison-agent \
  --dataset longmemeval \
  --track agent-oracle \
  --limit 10

# 3. Run an eval — UI
uv run unison-evals-server &              # FastAPI on :8001
cd web && bun install && bun dev          # Next.js on :3000
open http://localhost:3000/runs/new

How the comparison stays honest

  • Same production agent loop. The unison-agent adapter calls the /api/rest/agents/eval-turn endpoint, which runs the same runAgent loop that ships in production — retrieve → reason → answer, including the counting-verification gate (no eval-only forks in the answer path). Track 2 disables the brain/FS/workspace tools via the oracleContext request flag. One honest caveat: the eval seeds each question's brain synchronously (extractFromDocument → recordFact), which runs the same extraction logic as production but skips the asynchronous production ingestion pipeline (the signal notability gate, reconcile, and compaction). So the answering path is production; the brain-building path is a faster eval-time shortcut over identical extraction.
  • New brain-context contract. The unison-brain-context adapter reflects the post-restructure server that removed the agent endpoint. It is an SDK-customer emulation: provision → seed via /v1/eval/seed → GET /v1/brain/context → the harness runs a reader LLM over contextMd. The server never generates an answer — retrieval and generation are fully decoupled. See METHODOLOGY.md §6 for the full contract.
  • Fixed model + temperature. Judge model pinned per release (JUDGE_MODEL env var). All systems use temperature=0 where possible.
  • Fixed dataset versions. Datasets are downloaded from HuggingFace at a pinned commit hash and cached locally.
  • All numbers reproducible. Every run writes a JSON artifact with the exact dataset hash, model versions, timestamps, and per-question scores. Re-running the same config on the same hardware gets within ±2%.
  • Comparators run in their preferred config. Each adapter is configured per the system's docs. Issues / PRs welcome to fix any disadvantage we created accidentally.

What this benchmark is for (and what it isn't)

For: comparing memory/workspace agents on tasks that exercise persistent recall, multi-hop reasoning over knowledge, and bitemporal fact correctness.

Not for: comparing coding agents on shell tasks (use SWE-bench, Terminal-Bench) or computer-use agents on browser tasks (use WebArena, OSWorld). Unison isn't a coding agent — running it on SWE-bench would be a category mistake.

Architecture

┌──────────┐     fetch     ┌──────────────┐    asyncio    ┌──────────────┐
│ Next.js  │ ──────────→   │  FastAPI     │ ──────────→   │  Job worker  │
│  web UI  │ ←───── SSE    │  /api/runs/* │               │  (in-process)│
└──────────┘               └──────┬───────┘               └──────┬───────┘
                                  │                              │
                                  ↓                              ↓
                           ┌──────────────┐               ┌──────────────┐
                           │  SQLite      │               │  Adapters    │
                           │  (results)   │               │  (HTTP/CLI)  │
                           └──────────────┘               └──────────────┘

CLI calls the same job worker in-process (no HTTP). Server exists for the UI.

Adding a new system

One file, ~80 LOC. See src/unison_evals/memory_evals/adapters/base.py for the contract. PRs welcome.

License

Apache-2.0. Datasets retain their original licenses (LongMemEval is MIT, MemoryAgentBench is MIT).

Ver o original no GitHub

Bom para

Relacionados