claude-code-with-and-without-neo-mcp-customer-support-AI-agent-
dakshjain-1616/claude-code-with-and-without-neo-mcp-customer-support-AI-agent-
Instalação
npx convez-ai dakshjain-1616/claude-code-with-and-without-neo-mcp-customer-support-AI-agent-1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
README
A tradução para o português deste README ainda está na fila.
What Neo MCP adds to Claude Code
We gave Claude Code the exact same task twice. Once on its own. Once with Neo MCP running alongside it. The code both runs produced was line-for-line identical. The process, the documentation, and the reliability of how they got there was not.
This repo contains both runs so you can see for yourself.
The experiment
The task was to build a production-ready customer support AI agent from scratch using Claude's tool-use API. No starter code, no scaffolding, just a requirements brief.
The agent needed to handle FAQ retrieval through a TF-IDF knowledge base, create support tickets, escalate issues to human agents, and come with a full evaluation suite covering 50 structured test cases, 21 adversarial tests for prompt injection and jailbreak resistance, 19 hallucination tests for false premises and invented products, and a benchmark runner that ties it all together into a scored report.
Both runs used the same Claude model, the same Python dependencies, and the same CPU-only environment with no external embedding APIs.
Run 1 (neomcp/) used Claude Code with Neo MCP active.
Run 2 (claudecode/) used Claude Code on its own.
When both runs finished, the output was identical. 1,893 lines of Python across 9 files. Same architecture, same test counts, same structure. So what actually changed?
How the process played out
This is where the runs diverged immediately.
With Neo MCP, Claude Code started by building a plan. Not a mental note that got buried in the context window, but an actual file on disk. The plan mapped the full dependency chain in reverse: starting from "what does the final benchmark report need" and working backwards through every component until it reached the empty directory. That file lived at neomcp/plans/plan.md and served as the contract the rest of the run was held to.
Without Neo MCP, Claude Code reasoned through the same problem in its context window and started coding. The reasoning was solid. The plan was real. But it was never written anywhere.
That distinction turned out to matter a lot.
The context compaction problem
Claude Code keeps its working memory in the context window. On a focused, single-session task that fits in a few thousand tokens, this works fine. On a task spanning dozens of API calls, multiple architectural decisions, and nearly 2,000 lines of code, the window eventually fills up. When it does, the platform compacts it: older content gets summarized or dropped to make room for new content. The agent continues, but it continues without a full picture of what it has and has not done.
In the standalone run, this is what happened. The session compacted mid-task. When work resumed, the claudecode/ directory was empty. The agent had no record of any decisions made, no plan to refer back to, and no way to know how far along the implementation was. It had to read through all nine completed files in the neomcp/ folder just to figure out what it was supposed to be building, then copy and adapt from there.
This was not a crash. It was not a bug. It was Claude Code doing exactly what it is designed to do, running into a real limitation of context-window-only state management on a longer task.
Neo MCP sidesteps this entirely. Because state is written to disk at every step, it does not matter how many times the context window resets. The agent reads the directory, sees what exists, and continues from where it left off. No reconstruction, no re-reading completed work, no recovery tax.
Planning depth
The first question before writing any code was: what does this system actually need, and in what order should it be built?
| Dimension | With Neo MCP | Standalone |
|---|---|---|
| Persistent planning | plans/plan.md written before a single file was touched | Reasoned through in context, lost when the window compacted |
| API research | Model IDs, tool-use format, and message structure all confirmed upfront | Done, but never written down |
| Compute assessment | CPU and RAM limits confirmed before the architecture was chosen | Noted informally |
| Architecture docs | Dependency diagram and data flow written into implementation.md | Written, but after the fact |
| QA planning | A four-step verification cycle was designed before implementation started | No QA plan existed, checks happened reactively |
| Risk mitigation | Explicit upgrade paths documented for TF-IDF to embeddings and in-memory to database | Mentioned in passing |
The QA planning gap is the most consequential. In the Neo MCP run, the verification process was designed before the first file was written. In the standalone run, there was no QA plan at all. Checks happened after recovery, they were not structured, and nothing was recorded. The code still works, but if you want to know what was tested and what was not, there is no answer.
What the implementation actually looks like
Both runs produced the same code. The difference between them is in the scaffolding around that code: the planning artifact, the process documentation, the bug log, and the trace of decisions that were made along the way.
| With Neo MCP | Standalone | |
|---|---|---|
| Lines of code | 1,893 | 1,893 |
| Files produced | 9 | 9 |
| Planning artifact | plans/plan.md written before implementation started | None |
| QA process | py_compile each file, then import chain, then smoke tests, then JSON validation | Spot checks after recovery |
| Bugs caught and logged | 3 bugs found and fixed: BASE_DIR path resolution, JSON serialisation for numpy floats, thread safety in ticket store reset | Not recorded |
| Implementation doc length | 833 lines across 9 sections including a dedicated QA section | 393 lines, no QA section |
| Session continuity | Single uninterrupted session from start to finish | Context compaction required a full recovery pass |
A note on bugs: the three issues caught during the Neo MCP run were real problems that would have caused failures. The BASE_DIR fix ensures eval_cases.json is found regardless of which directory the benchmark is run from. The JSON serialisation fix prevents numpy.float32 cosine similarity scores from crashing the tool result serialisation. The thread safety fix ensures the ticket counter resets atomically alongside the store. All three are documented in neomcp/customer_support_agent/implementation.md. Whether the same issues exist in the standalone run and were simply not caught, or were caught and not recorded, is impossible to say.
The documentation gap
Both runs produced an implementation.md file. Here is what those files actually contain:
The most striking gap is the QA section. The Neo MCP implementation doc has 80 lines on the verification process alone: the exact commands run, what each command checks for, and the three bugs that were caught and fixed. The standalone doc has no equivalent section at all.
This is not just a documentation preference. In a team setting, the question "was this tested?" only has a meaningful answer if someone wrote down what they tested and what they found. A QA section that says "we ran python3 -m py_compile on every file and found a JSON serialisation issue in agent.py" is audit evidence. A QA section that does not exist is not.
The same pattern holds across other sections. The tradeoff analysis in the Neo MCP doc includes structured comparison tables for TF-IDF versus dense embeddings, in-memory storage versus SQLite, and different confidence threshold values, each with an explicit upgrade path. The standalone doc covers the same decisions in prose, which is readable but harder to act on.
Scorecard
The +33 point gap comes almost entirely from three dimensions: session continuity (+8), upfront planning (+7), and no recovery required (+7). These are all downstream effects of the same root cause: whether task state lives on disk or in the context window.
Why this gap exists
It is worth being precise about what Neo MCP actually does here, because the answer is not that it makes Claude smarter or that it writes better code.
Claude Code is a capable coding agent. It can reason through complex architecture, write clean Python, and catch its own bugs. What it cannot do on its own is guarantee that the reasoning and decisions it produces will still be accessible an hour later when the context window has moved on.
Neo MCP adds a workflow layer on top of Claude Code. That layer does a few specific things. It prompts the agent to produce planning artifacts before starting implementation, so intent is captured in files rather than in context. It structures execution into discrete, verifiable phases so each step can be confirmed before the next begins. And it ensures that when a session ends or a context window compacts, the task can resume from disk state rather than from inference.
The result is not smarter code. It is a more reliable process for producing the same code, with a paper trail that shows how you got there.
A closer look at the knowledge base
The core of the agent is a TF-IDF knowledge base with 35 FAQ entries covering eight support categories. Here is how the retrieval performs on a few representative queries:
| Query | Top result category | Confidence score |
|---|---|---|
| "reset my password" | password | 0.426 |
| "billing history" | billing | 0.550 |
| "refund policy" | refund | 0.490 |
| "track my order" | shipping | 0.480 |
| "what is the meaning of life" | (none matched) | 0.000 |
The confidence threshold is set at 0.30. Anything above that gets answered from the FAQ. Anything below triggers an offer to escalate or create a ticket. The last row is an edge case test from the evaluation suite: the agent is supposed to politely decline and stay in scope, which it does.
The evaluation suite in detail
The 50 evaluation cases are distributed across nine categories, each chosen to test a different kind of agent behavior:
| Category | Cases | What is being tested |
|---|---|---|
| Billing | 8 | FAQ retrieval for common questions, ticket creation for disputes |
| Account | 8 | Standard FAQ, plus escalation for unauthorized access |
| Technical | 7 | Browser support and API docs from FAQ, ticket creation for active bugs |
| Password | 5 | Self-service flows from FAQ, ticket for account recovery edge cases |
| Refund | 5 | Policy questions answered from FAQ |
| Shipping | 5 | Tracking, international, damage claims all in FAQ |
| Products | 5 | General product questions from FAQ, tickets for detailed feature requests |
| Policy | 4 | Privacy and data retention questions from FAQ |
| Edge cases | 3 | Out-of-scope deflection and general conversational handling |
The adversarial suite covers prompt injection attempts, jailbreak patterns like DAN mode and character roleplay bypasses, social engineering such as admin impersonation, and encoded inputs including base64, SQL injection patterns, and Unicode homoglyph spoofing. The hallucination suite tests whether the agent invents answers for non-existent products, false premises like a "75% student discount" that does not exist, and specific numbers that are not in the knowledge base.
Summary
| With Neo MCP | Without Neo MCP | |
|---|---|---|
| Code produced | 1,893 lines, 9 files | 1,893 lines, 9 files |
| Planning artifact on disk | Yes, written before implementation | No, lost when context compacted |
| Session continuity | Single uninterrupted session | Context compaction required recovery |
| QA process | Four-step cycle, three bugs logged | Reactive spot checks, nothing recorded |
| State management | Disk-backed, recoverable at any point | Context-only, gone when the window fills |
| Documentation depth | 833 lines including QA and bug log | 393 lines, no QA section |
| Overall score | 96 out of 100 | 63 out of 100 |
The same AI. The same task. The code that came out was identical. Neo MCP is the difference between a process you can audit and repeat, and one that works well when nothing goes wrong.
Repository structure
task1/
├── README.md
├── assets/
│ ├── deliverables.svg
│ ├── process_flow.svg
│ ├── planning_radar.svg
│ ├── doc_depth.svg
│ ├── scorecard.svg
│ └── state_management.svg
│
├── neomcp/ Run 1: Claude Code with Neo MCP
│ ├── plans/
│ │ └── plan.md upfront dependency-chain plan
│ └── customer_support_agent/
│ ├── knowledge_base.py
│ ├── tools.py
│ ├── agent.py
│ ├── eval_cases.json
│ ├── adversarial_tests.py
│ ├── hallucination_tests.py
│ ├── benchmark_runner.py
│ ├── requirements.txt
│ ├── README.md
│ └── implementation.md 833 lines, includes QA section and bug log
│
└── claudecode/ Run 2: Claude Code standalone
└── customer_support_agent/
├── knowledge_base.py
├── tools.py
├── agent.py
├── eval_cases.json
├── adversarial_tests.py
├── hallucination_tests.py
├── benchmark_runner.py
├── requirements.txt
├── README.md
└── implementation.md 393 lines, no QA section
Running either implementation
cd neomcp/customer_support_agent
pip install -r requirements.txt
export ANTHROPIC_API_KEY='your-key-here'
python benchmark_runner.py
The same commands work from claudecode/customer_support_agent/. Both implementations are fully functional and produce the same benchmark output.