convez.ai
MCP Server
dakshjain-1616

claude-code-with-and-without-neo-mcp-customer-support-AI-agent-

dakshjain-1616/claude-code-with-and-without-neo-mcp-customer-support-AI-agent-

Ver no GitHub
InicialPluginScore · de 0 a 100

Instalação

$npx convez-ai dakshjain-1616/claude-code-with-and-without-neo-mcp-customer-support-AI-agent-

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
1
Forks
0
Acompanhando
0
Saúde
14%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

README

A tradução para o português deste README ainda está na fila.

What Neo MCP adds to Claude Code

We gave Claude Code the exact same task twice. Once on its own. Once with Neo MCP running alongside it. The code both runs produced was line-for-line identical. The process, the documentation, and the reliability of how they got there was not.

This repo contains both runs so you can see for yourself.


The experiment

The task was to build a production-ready customer support AI agent from scratch using Claude's tool-use API. No starter code, no scaffolding, just a requirements brief.

The agent needed to handle FAQ retrieval through a TF-IDF knowledge base, create support tickets, escalate issues to human agents, and come with a full evaluation suite covering 50 structured test cases, 21 adversarial tests for prompt injection and jailbreak resistance, 19 hallucination tests for false premises and invented products, and a benchmark runner that ties it all together into a scored report.

Both runs used the same Claude model, the same Python dependencies, and the same CPU-only environment with no external embedding APIs.

Run 1 (neomcp/) used Claude Code with Neo MCP active.
Run 2 (claudecode/) used Claude Code on its own.

Deliverables

When both runs finished, the output was identical. 1,893 lines of Python across 9 files. Same architecture, same test counts, same structure. So what actually changed?


How the process played out

This is where the runs diverged immediately.

With Neo MCP, Claude Code started by building a plan. Not a mental note that got buried in the context window, but an actual file on disk. The plan mapped the full dependency chain in reverse: starting from "what does the final benchmark report need" and working backwards through every component until it reached the empty directory. That file lived at neomcp/plans/plan.md and served as the contract the rest of the run was held to.

Without Neo MCP, Claude Code reasoned through the same problem in its context window and started coding. The reasoning was solid. The plan was real. But it was never written anywhere.

That distinction turned out to matter a lot.

Process Flow

The context compaction problem

Claude Code keeps its working memory in the context window. On a focused, single-session task that fits in a few thousand tokens, this works fine. On a task spanning dozens of API calls, multiple architectural decisions, and nearly 2,000 lines of code, the window eventually fills up. When it does, the platform compacts it: older content gets summarized or dropped to make room for new content. The agent continues, but it continues without a full picture of what it has and has not done.

In the standalone run, this is what happened. The session compacted mid-task. When work resumed, the claudecode/ directory was empty. The agent had no record of any decisions made, no plan to refer back to, and no way to know how far along the implementation was. It had to read through all nine completed files in the neomcp/ folder just to figure out what it was supposed to be building, then copy and adapt from there.

This was not a crash. It was not a bug. It was Claude Code doing exactly what it is designed to do, running into a real limitation of context-window-only state management on a longer task.

Neo MCP sidesteps this entirely. Because state is written to disk at every step, it does not matter how many times the context window resets. The agent reads the directory, sees what exists, and continues from where it left off. No reconstruction, no re-reading completed work, no recovery tax.


Planning depth

The first question before writing any code was: what does this system actually need, and in what order should it be built?

Planning Radar

DimensionWith Neo MCPStandalone
Persistent planningplans/plan.md written before a single file was touchedReasoned through in context, lost when the window compacted
API researchModel IDs, tool-use format, and message structure all confirmed upfrontDone, but never written down
Compute assessmentCPU and RAM limits confirmed before the architecture was chosenNoted informally
Architecture docsDependency diagram and data flow written into implementation.mdWritten, but after the fact
QA planningA four-step verification cycle was designed before implementation startedNo QA plan existed, checks happened reactively
Risk mitigationExplicit upgrade paths documented for TF-IDF to embeddings and in-memory to databaseMentioned in passing

The QA planning gap is the most consequential. In the Neo MCP run, the verification process was designed before the first file was written. In the standalone run, there was no QA plan at all. Checks happened after recovery, they were not structured, and nothing was recorded. The code still works, but if you want to know what was tested and what was not, there is no answer.


What the implementation actually looks like

Both runs produced the same code. The difference between them is in the scaffolding around that code: the planning artifact, the process documentation, the bug log, and the trace of decisions that were made along the way.

With Neo MCPStandalone
Lines of code1,8931,893
Files produced99
Planning artifactplans/plan.md written before implementation startedNone
QA processpy_compile each file, then import chain, then smoke tests, then JSON validationSpot checks after recovery
Bugs caught and logged3 bugs found and fixed: BASE_DIR path resolution, JSON serialisation for numpy floats, thread safety in ticket store resetNot recorded
Implementation doc length833 lines across 9 sections including a dedicated QA section393 lines, no QA section
Session continuitySingle uninterrupted session from start to finishContext compaction required a full recovery pass

A note on bugs: the three issues caught during the Neo MCP run were real problems that would have caused failures. The BASE_DIR fix ensures eval_cases.json is found regardless of which directory the benchmark is run from. The JSON serialisation fix prevents numpy.float32 cosine similarity scores from crashing the tool result serialisation. The thread safety fix ensures the ticket counter resets atomically alongside the store. All three are documented in neomcp/customer_support_agent/implementation.md. Whether the same issues exist in the standalone run and were simply not caught, or were caught and not recorded, is impossible to say.


The documentation gap

Both runs produced an implementation.md file. Here is what those files actually contain:

Documentation Depth

The most striking gap is the QA section. The Neo MCP implementation doc has 80 lines on the verification process alone: the exact commands run, what each command checks for, and the three bugs that were caught and fixed. The standalone doc has no equivalent section at all.

This is not just a documentation preference. In a team setting, the question "was this tested?" only has a meaningful answer if someone wrote down what they tested and what they found. A QA section that says "we ran python3 -m py_compile on every file and found a JSON serialisation issue in agent.py" is audit evidence. A QA section that does not exist is not.

The same pattern holds across other sections. The tradeoff analysis in the Neo MCP doc includes structured comparison tables for TF-IDF versus dense embeddings, in-memory storage versus SQLite, and different confidence threshold values, each with an explicit upgrade path. The standalone doc covers the same decisions in prose, which is readable but harder to act on.


Scorecard

Scorecard

The +33 point gap comes almost entirely from three dimensions: session continuity (+8), upfront planning (+7), and no recovery required (+7). These are all downstream effects of the same root cause: whether task state lives on disk or in the context window.


Why this gap exists

It is worth being precise about what Neo MCP actually does here, because the answer is not that it makes Claude smarter or that it writes better code.

Claude Code is a capable coding agent. It can reason through complex architecture, write clean Python, and catch its own bugs. What it cannot do on its own is guarantee that the reasoning and decisions it produces will still be accessible an hour later when the context window has moved on.

State Management

Neo MCP adds a workflow layer on top of Claude Code. That layer does a few specific things. It prompts the agent to produce planning artifacts before starting implementation, so intent is captured in files rather than in context. It structures execution into discrete, verifiable phases so each step can be confirmed before the next begins. And it ensures that when a session ends or a context window compacts, the task can resume from disk state rather than from inference.

The result is not smarter code. It is a more reliable process for producing the same code, with a paper trail that shows how you got there.


A closer look at the knowledge base

The core of the agent is a TF-IDF knowledge base with 35 FAQ entries covering eight support categories. Here is how the retrieval performs on a few representative queries:

QueryTop result categoryConfidence score
"reset my password"password0.426
"billing history"billing0.550
"refund policy"refund0.490
"track my order"shipping0.480
"what is the meaning of life"(none matched)0.000

The confidence threshold is set at 0.30. Anything above that gets answered from the FAQ. Anything below triggers an offer to escalate or create a ticket. The last row is an edge case test from the evaluation suite: the agent is supposed to politely decline and stay in scope, which it does.


The evaluation suite in detail

The 50 evaluation cases are distributed across nine categories, each chosen to test a different kind of agent behavior:

CategoryCasesWhat is being tested
Billing8FAQ retrieval for common questions, ticket creation for disputes
Account8Standard FAQ, plus escalation for unauthorized access
Technical7Browser support and API docs from FAQ, ticket creation for active bugs
Password5Self-service flows from FAQ, ticket for account recovery edge cases
Refund5Policy questions answered from FAQ
Shipping5Tracking, international, damage claims all in FAQ
Products5General product questions from FAQ, tickets for detailed feature requests
Policy4Privacy and data retention questions from FAQ
Edge cases3Out-of-scope deflection and general conversational handling

The adversarial suite covers prompt injection attempts, jailbreak patterns like DAN mode and character roleplay bypasses, social engineering such as admin impersonation, and encoded inputs including base64, SQL injection patterns, and Unicode homoglyph spoofing. The hallucination suite tests whether the agent invents answers for non-existent products, false premises like a "75% student discount" that does not exist, and specific numbers that are not in the knowledge base.


Summary

With Neo MCPWithout Neo MCP
Code produced1,893 lines, 9 files1,893 lines, 9 files
Planning artifact on diskYes, written before implementationNo, lost when context compacted
Session continuitySingle uninterrupted sessionContext compaction required recovery
QA processFour-step cycle, three bugs loggedReactive spot checks, nothing recorded
State managementDisk-backed, recoverable at any pointContext-only, gone when the window fills
Documentation depth833 lines including QA and bug log393 lines, no QA section
Overall score96 out of 10063 out of 100

The same AI. The same task. The code that came out was identical. Neo MCP is the difference between a process you can audit and repeat, and one that works well when nothing goes wrong.


Repository structure

task1/
├── README.md
├── assets/
│   ├── deliverables.svg
│   ├── process_flow.svg
│   ├── planning_radar.svg
│   ├── doc_depth.svg
│   ├── scorecard.svg
│   └── state_management.svg
│
├── neomcp/                              Run 1: Claude Code with Neo MCP
│   ├── plans/
│   │   └── plan.md                      upfront dependency-chain plan
│   └── customer_support_agent/
│       ├── knowledge_base.py
│       ├── tools.py
│       ├── agent.py
│       ├── eval_cases.json
│       ├── adversarial_tests.py
│       ├── hallucination_tests.py
│       ├── benchmark_runner.py
│       ├── requirements.txt
│       ├── README.md
│       └── implementation.md            833 lines, includes QA section and bug log
│
└── claudecode/                          Run 2: Claude Code standalone
    └── customer_support_agent/
        ├── knowledge_base.py
        ├── tools.py
        ├── agent.py
        ├── eval_cases.json
        ├── adversarial_tests.py
        ├── hallucination_tests.py
        ├── benchmark_runner.py
        ├── requirements.txt
        ├── README.md
        └── implementation.md            393 lines, no QA section

Running either implementation

cd neomcp/customer_support_agent
pip install -r requirements.txt
export ANTHROPIC_API_KEY='your-key-here'
python benchmark_runner.py

The same commands work from claudecode/customer_support_agent/. Both implementations are fully functional and produce the same benchmark output.

Ver o original no GitHub

Bom para

Relacionados