mcp-usability-test
jeffgrover/mcp-usability-test
Dual-agent harness for testing MCP server usability
Instalação
npx convez-ai jeffgrover/mcp-usability-test1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
README
A tradução para o português deste README ainda está na fila.
MCP Usability Test
A dual-agent harness for testing whether an MCP server is pleasant, discoverable, and recoverable for an AI agent to use.
One local LLM plays the User and calls MCP tools. A second local LLM plays the Tester, assigns tasks, watches the transcript, and records structured UX observations. Functional tests tell you whether a tool works. This tells you whether an agent can figure out how to use it.
What It Tests
- Tool discoverability and naming
- Parameter clarity
- Error messages and recovery paths
- Result formats
- Workflow efficiency across multiple tools
- Missing capabilities surfaced by realistic tasks
Quick Demo: You.com Search MCP Server
The example config runs the You.com MCP server over stdio in free search-only mode. No API key is required for the first demo.
Requirements:
- Python 3.11+
- Node.js and
npx - LM Studio running an OpenAI-compatible local server on
localhost:1234
Setup:
git clone https://github.com/jeffgrover/mcp-usability-test.git
cd mcp-usability-test
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
cp config.example.yaml config.yaml
Optional: set YDC_API_KEY in your local environment if you want higher
You.com rate limits. Keep local key files such as .env out of git; it is
ignored by default.
Start LM Studio, load a tool-calling model, enable the local server, then run:
python -m src.main
The first run uses the scenarios in scenarios/you_search/ to evaluate basic
search, source comparison, recent-information behavior, and error recovery.
Configuration
Copy config.example.yaml to config.yaml and edit locally. config.yaml is
gitignored because it may contain local URLs, model names, and credentials.
The default MCP section demonstrates stdio:
mcp_server:
transport: "stdio"
command: "npx"
args:
- "-y"
- "@youdotcom-oss/mcp"
env:
YDC_PROFILE: "free"
oauth:
enabled: false
For HTTP or SSE MCP servers, switch transports and use headers:
mcp_server:
transport: "streamable_http"
url: "https://mcp.example.com/mcp"
headers:
Authorization: "Bearer ${MCP_API_KEY}"
oauth:
enabled: false
OAuth remains available for servers that need a browser authorization flow:
mcp_server:
transport: "streamable_http"
url: "https://mcp.example.com/mcp"
oauth:
enabled: true
callback_port: 8100
scopes: ""
token_file: "state/oauth_tokens.json"
Running
# Run configured scenarios from the beginning
python -m src.main --fresh
# Resume saved state
python -m src.main --resume
# Free exploration mode: keep probing tools until you stop it
python -m src.main --explore
# Use another config file
python -m src.main -c my-config.yaml
# Verbose logs
python -m src.main -v
Press Ctrl+C at any time. Session state is saved after every round.
How Scenarios Work
Scenarios are YAML files with a persona, task goals, and tester focus areas. They do not script exact tool calls. The Tester asks the User to accomplish a task naturally, then records what made the tool interface easy or difficult.
Example:
name: "Basic Search"
description: "Evaluate first-run search usability."
persona: >
You are a curious technical user trying to answer a question with the
available MCP tools.
eval_goals:
- id: discover_tools
task: "Figure out which search tools are available."
success_hint: "The User identifies a plausible search tool."
tester_focus:
- "Are tool names and parameter names obvious?"
- "Are results easy to interpret?"
max_rounds: 6
Output
Generated artifacts are written locally and ignored by git:
observations/session_*.md- structured UX findingsstate/session.json- resumable session statestate/transcript.jsonl- complete conversation and tool transcriptharness.log- verbose logs when enabled
Architecture
Tester agent <---- orchestrator ----> User agent
| |
| observes transcript | calls tools
v v
observations/*.md MCP server
The User sees tool definitions and can call tools. The Tester sees tool calls and results, but never calls tools directly. The Tester's observations are stripped before messages are shown to the User.
Supported MCP transports:
stdiostreamable_httpsse
Supported auth patterns:
- Environment variables for stdio servers
- Headers for HTTP/SSE servers
- OAuth with PKCE and local callback token storage
Development Checks
python -m compileall -q src
python -m src.main -c config.example.yaml --fresh
The second command requires LM Studio, Node.js, and network access for the
first npx package fetch.
License
MIT