convez.ai
MCP ServerDados & AnalyticsAvançado
PythonicVarun

LLM Chess Evaluation Harness

PythonicVarun/llm-chess-evaluation-harness

Avalie LLMs jogando xadrez contra Stockfish com dashboard em tempo real

Ver no GitHub
InicialPluginScore · de 0 a 100

Framework de avaliação para testar capacidades de xadrez de LLMs contra Stockfish através de um servidor MCP. Fornece interface TUI em tempo real, validação robusta de movimentos e exportação PGN automática, suportando qualquer endpoint compatível com OpenAI.

Instalação

$npx convez-ai PythonicVarun/llm-chess-evaluation-harness

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Avaliar habilidades de raciocínio estratégico de LLMs
  • Benchmarking de modelos contra diferentes níveis de dificuldade
  • Análise de falhas e padrões de erros em tomada de decisão

Veredito

Ferramenta bem arquitetada para benchmark objetivo de LLMs em domínio específico. Combina MCP server robusto com interface polida; ideal para pesquisadores avaliando raciocínio tático-estratégico de modelos.

README

A tradução para o português deste README ainda está na fila.

♟️ LLM Chess Evaluation Harness

A sophisticated evaluation framework for testing the chess-playing capabilities of Large Language Models (LLMs) against Stockfish. 🤖 The harness uses a Model Context Protocol (MCP) server to maintain secure, stateful chess games and provides a rich live-updating terminal interface. 🖥️

It supports any OpenAI-compatible API endpoint, allowing you to test OpenAI models, Anthropic models (via proxies), or local LLMs (via vLLM, Ollama, etc.). 🌐

✨ Features

  • 📺 Live Terminal Dashboard: A beautiful, real-time TUI built with rich, displaying the board, move history, side-to-move, and live game status.
  • 🛠️ MCP Backend: Chess state and rules are fully mediated by a local MCP server (chess_mcp_server.py), providing robust move validation and state management.
  • 🐟 Stockfish Integration: Test against Stockfish with configurable ELO levels to find the exact rating of your LLM.
  • 🛡️ Robust Error Handling: Automatically detects illegal LLM moves and prompts the LLM to retry, tracking "illegal move attempts" as a metric.
  • 💾 PGN Export: Automatically saves all finished games to standard Portable Game Notation (PGN) files for later review in any chess GUI. Filenames include the model id and Stockfish ELO for easy sorting.

📋 Requirements

  • 🐍 Python 3.12+
  • ⚙️ Stockfish: The Stockfish chess engine binary must be installed on your system.
    • 🐧 Ubuntu/Debian: sudo apt install stockfish
    • 🍎 macOS: brew install stockfish
    • 🪟 Windows: Download from the Stockfish website and add it to your PATH, or specify the path via configuration.
  • 🔑 API Key: An API key for your chosen LLM provider (e.g., OpenAI).

🚀 Installation

  1. 📥 Clone this repository.
  2. 📦 Install the required dependencies:
pip install -r requirements.txt
# or using uv:
# uv sync
  1. 🔧 Configure your environment variables. You can create a .env file in the project root:
LLM_API_KEY=your_api_key_here
# Optional:
# LLM_BASE_URL=https://api.your-custom-provider.com/v1
# STOCKFISH_PATH=/path/to/custom/stockfish

⚡ Direct Execution (No Cloning/Installation Required)

You can run the harness or the MCP server directly from the GitHub repository using uv or uvx without cloning the project.

1. Using uv run

Run the evaluation harness command directly:

uv run --with git+https://github.com/PythonicVarun/llm-chess-evaluation-harness.git eval [options]

2. Using uvx (or uv tool run)

Run the evaluation harness:

uvx --from git+https://github.com/PythonicVarun/llm-chess-evaluation-harness.git eval [options]

Or start the standalone MCP Chess server:

uvx --from git+https://github.com/PythonicVarun/llm-chess-evaluation-harness.git mcp

[!NOTE] Make sure your OPENAI_API_KEY (or LLM_API_KEY) and STOCKFISH_PATH environment variables are exported in your terminal before running the direct command.

🎮 Usage

Run the evaluation harness using the main script:

python chess_eval.py

By default, this will play a 3-game match using gpt-4o-mini as White against Stockfish (1500 ELO) as Black. ⚔️

🎛️ Command Line Arguments

You can override the default configuration using CLI arguments:

  • --model <name>: 🧠 LLM model name (default: gpt-4o-mini).
  • --reasoning-effort <low|medium|high>: 🧠 Reasoning effort for supported models.
  • --color <white|black>: 🎨 The color the LLM will play (default: white).
  • --games <n>: 🔢 Number of games to play in the match (default: 3).
  • --elo <n>: 📈 Stockfish target ELO (default: 1320).
  • --base-url <url>: 🔗 Custom OpenAI-compatible endpoint URL.
  • --api-key <key>: 🗝️ Override the API key explicitly.
  • --stockfish <path>: 📍 Path to the Stockfish binary.
  • --output-dir <path>: 📁 Directory to save PGN files (default: pgn_output).
  • --temperature <n>: 🌡️ Model temperature (default: 0.2).
  • --log-level <DEBUG|INFO|WARNING|ERROR>: 📝 Set internal logging verbosity (saved to logs/<model>_elo<stockfish-elo>.log).

💡 Example: Testing a local model via vLLM

If you are running a local model server compatible with the OpenAI API:

python chess_eval.py --model meta-llama/Meta-Llama-3-8B-Instruct --reasoning-effort medium --base-url http://localhost:8000/v1 --games 5 --elo 1320

📊 Evaluation Results

Below is a summary of the matches played between various LLM models and Stockfish at different target ELO levels. The table shows the number of wins, losses, draws, total games played, and the resulting win rate for each model at each ELO level.

ModelReasoningStockfish ELOWinsLossesDrawsTotal GamesWin Rate
gemini-3.5-flashdefault200003030.0%
gpt-5.5none200003030.0%
gemini-3.5-flashdefault190002130.0%
gpt-5.5none190002130.0%
gemini-3.5-flashdefault1800111333.3%
gpt-5.5none170002130.0%
gpt-5.5low170003030.0%
gemini-3.5-flashdefault1600120333.3%
gemini-3.5-flashdefault1500210366.7%
gpt-5.4-minihigh150002020.0%
gpt-5.4-nanodefault150002020.0%
gpt-5.4-nanonone150003030.0%
gpt-5.4-nanolow150003030.0%
gpt-5.4-nanomedium150003030.0%
gpt-5.5none150002130.0%
gpt-5.5low1500111333.3%
gemini-3.5-flashdefault1450120333.3%
gemini-3.5-flashdefault1400111333.3%
gemini-3.1-flash-litedefault132003030.0%
gemini-3.5-flashdefault1320210366.7%
gpt-4.1default132003030.0%
gpt-4.1-minidefault132003030.0%
gpt-4.1-nanodefault132003030.0%
gpt-4odefault132003030.0%
gpt-4o-minidefault132003030.0%
gpt-5.4default132003030.0%
gpt-5.4-minidefault132003030.0%
gpt-5.4-mininone132003030.0%
gpt-5.4-minilow132003030.0%
gpt-5.4-minimedium132003030.0%
gpt-5.4-minihigh132002130.0%
gpt-5.4-nanodefault132006060.0%
gpt-5.4-nanonone132002130.0%
gpt-5.4-nanolow132003030.0%
gpt-5.4-nanomedium132004260.0%
gpt-5.4-nanohigh132003030.0%
gpt-5.5default13203003100.0%
gpt-5.5none132001120.0%
gpt-5.5low1320111333.3%
gpt-5.5medium1320201366.7%
gpt-5.5high13201001100.0%

🪙 Token & Cost Summarization

At the end of each evaluation match, the harness prints a detailed Token Usage & Cost Summary in your terminal. This shows the total prompt (input) tokens, completion (output) tokens, and the estimated total cost in USD for the model run.

Model costs are independently stored in eval_config.py and are based on the following official developer pricing guides:

For custom or local models where pricing is not known, the estimated cost defaults to $0.00000. You can configure or override custom model costs by editing the model_pricing dictionary in eval_config.py or overriding it in your custom code.

🏗️ Architecture

  • 🎬 chess_eval.py: The orchestrator and UI layer. It starts the MCP server, initializes Stockfish, queries the LLM for moves, and updates the dashboard.
  • ⚙️ chess_mcp_server.py: A stateless script running as an MCP server over stdio. It wraps the python-chess library, exposing tools to read the board (get_board_state), validate moves (validate_move), apply moves (make_move), and export the game (export_pgn).
  • 📝 eval_config.py: Centralized dataclass for evaluation settings and defaults.

📜 License

This project is licensed under the MIT License. ⚖️

Ver o original no GitHub

Bom para

Relacionados