convez.ai
MCP Server
jeffgrover

mcp-usability-test

jeffgrover/mcp-usability-test

Ver no GitHub
InicialPluginScore · de 0 a 100

Dual-agent harness for testing MCP server usability

Instalação

$npx convez-ai jeffgrover/mcp-usability-test

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

README

A tradução para o português deste README ainda está na fila.

MCP Usability Test

A dual-agent harness for testing whether an MCP server is pleasant, discoverable, and recoverable for an AI agent to use.

One local LLM plays the User and calls MCP tools. A second local LLM plays the Tester, assigns tasks, watches the transcript, and records structured UX observations. Functional tests tell you whether a tool works. This tells you whether an agent can figure out how to use it.

What It Tests

  • Tool discoverability and naming
  • Parameter clarity
  • Error messages and recovery paths
  • Result formats
  • Workflow efficiency across multiple tools
  • Missing capabilities surfaced by realistic tasks

Quick Demo: You.com Search MCP Server

The example config runs the You.com MCP server over stdio in free search-only mode. No API key is required for the first demo.

Requirements:

  • Python 3.11+
  • Node.js and npx
  • LM Studio running an OpenAI-compatible local server on localhost:1234

Setup:

git clone https://github.com/jeffgrover/mcp-usability-test.git
cd mcp-usability-test
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
cp config.example.yaml config.yaml

Optional: set YDC_API_KEY in your local environment if you want higher You.com rate limits. Keep local key files such as .env out of git; it is ignored by default.

Start LM Studio, load a tool-calling model, enable the local server, then run:

python -m src.main

The first run uses the scenarios in scenarios/you_search/ to evaluate basic search, source comparison, recent-information behavior, and error recovery.

Configuration

Copy config.example.yaml to config.yaml and edit locally. config.yaml is gitignored because it may contain local URLs, model names, and credentials.

The default MCP section demonstrates stdio:

mcp_server:
  transport: "stdio"
  command: "npx"
  args:
    - "-y"
    - "@youdotcom-oss/mcp"
  env:
    YDC_PROFILE: "free"
  oauth:
    enabled: false

For HTTP or SSE MCP servers, switch transports and use headers:

mcp_server:
  transport: "streamable_http"
  url: "https://mcp.example.com/mcp"
  headers:
    Authorization: "Bearer ${MCP_API_KEY}"
  oauth:
    enabled: false

OAuth remains available for servers that need a browser authorization flow:

mcp_server:
  transport: "streamable_http"
  url: "https://mcp.example.com/mcp"
  oauth:
    enabled: true
    callback_port: 8100
    scopes: ""
    token_file: "state/oauth_tokens.json"

Running

# Run configured scenarios from the beginning
python -m src.main --fresh

# Resume saved state
python -m src.main --resume

# Free exploration mode: keep probing tools until you stop it
python -m src.main --explore

# Use another config file
python -m src.main -c my-config.yaml

# Verbose logs
python -m src.main -v

Press Ctrl+C at any time. Session state is saved after every round.

How Scenarios Work

Scenarios are YAML files with a persona, task goals, and tester focus areas. They do not script exact tool calls. The Tester asks the User to accomplish a task naturally, then records what made the tool interface easy or difficult.

Example:

name: "Basic Search"
description: "Evaluate first-run search usability."
persona: >
  You are a curious technical user trying to answer a question with the
  available MCP tools.
eval_goals:
  - id: discover_tools
    task: "Figure out which search tools are available."
    success_hint: "The User identifies a plausible search tool."
tester_focus:
  - "Are tool names and parameter names obvious?"
  - "Are results easy to interpret?"
max_rounds: 6

Output

Generated artifacts are written locally and ignored by git:

  • observations/session_*.md - structured UX findings
  • state/session.json - resumable session state
  • state/transcript.jsonl - complete conversation and tool transcript
  • harness.log - verbose logs when enabled

Architecture

Tester agent  <---- orchestrator ---->  User agent
    |                                      |
    | observes transcript                  | calls tools
    v                                      v
observations/*.md                    MCP server

The User sees tool definitions and can call tools. The Tester sees tool calls and results, but never calls tools directly. The Tester's observations are stripped before messages are shown to the User.

Supported MCP transports:

  • stdio
  • streamable_http
  • sse

Supported auth patterns:

  • Environment variables for stdio servers
  • Headers for HTTP/SSE servers
  • OAuth with PKCE and local callback token storage

Development Checks

python -m compileall -q src
python -m src.main -c config.example.yaml --fresh

The second command requires LM Studio, Node.js, and network access for the first npx package fetch.

License

MIT

Ver o original no GitHub

Relacionados