convez.ai
MCP ServerAutomaçãoAvançado
ImagineTheGames

UnrealAgentPlayer

ImagineTheGames/UnrealAgentPlayer

Deixe agentes de IA testar Unreal automaticamente — compile, jogue, dirija, valide

Ver no GitHub
InicialPluginScore · de 0 a 100

MCP server + plugin C++ para Unreal Engine que permite agentes de IA dirigir o editor (PIE, injeção de input, logs, estado, screenshots) em loop fechado de teste autônomo. Integra-se com Claude Code, Claude Desktop e Cursor para validação automática de mudanças de gameplay sem intervenção humana.

Instalação

$npx convez-ai ImagineTheGames/UnrealAgentPlayer

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
5
Forks
2
Acompanhando
0
Saúde
42%
Último commit
há 1 dia
Versão
—
Issues abertas
1
Contribuidores
1

Casos de uso

  • Testes automatizados de gameplay e mecânicas em C++/Blueprints
  • Validação de mudanças de código direto no PIE com screenshots e logs
  • Ciclos de desenvolvimento fechados: edit → compile → test → assert

Veredito

Solução especializada de alto nível técnico que fecha um gap real: agentes de IA agora conseguem validar gameplay após compilação, não apenas escrever código. Excelente para pipelines de desenvolvimento agentic em UE, mas requer Windows, conhecimento profundo de Unreal e arquitetura MCP.

README

A tradução para o português deste README ainda está na fila.

UnrealAgentPlayer

Let an AI coding agent drive the Unreal Editor — start Play-In-Editor, inject real input, read logs and game state, capture screenshots, and run project-specific test helpers — to close the loop on its own changes.

UnrealAgentPlayer is a thin C++ editor plugin plus an external Python MCP server. Together they give an agent (Claude Code, Claude Desktop, Cursor, or any MCP client) a structured, scriptable handle on a running editor: make a change, compile it, play it, drive it with keyboard/mouse/gamepad/VR input, observe what happened, and decide what to do next — without a human pressing keys.

  • Plugin (Plugin/): a UEditorSubsystem that exposes a handful of UFUNCTIONs to UE's built-in Remote Control, plus log capture, input injection, a fake motion-controller modular feature, and Blueprint test-helper discovery.
  • MCP server (mcp-server/): a stdio MCP server that talks to the editor over Remote Control HTTP and Python Remote Execution, and presents ~30 agent-facing tools across 12 families. The Python layer grows new tools without recompiling the engine.

Supports UE 5.6, 5.7, and the Meta 5.7 fork. Windows editor only (see Platform support).

Status: Functional and runtime-proven on real projects (see Proven). Versioned 0.0.1; API may still change.


Why

Agentic coding tools can write and compile UE C++ and Blueprints, but they're usually blind after the build: they can't tell whether the thing they changed actually works in play. Verifying a gameplay change normally means a human launches PIE, presses some keys, and eyeballs the result.

UnrealAgentPlayer turns that manual loop into an API:

edit → compile → pie_start → inject input → read logs / state / screenshot → assert → pie_stop

The agent perceives and acts inside the editor the way a tester would, so "does this actually work?" becomes a thing it can answer itself.


Architecture

  MCP client (Claude Code / Desktop / Cursor)
        │  stdio (MCP)
        ▼
  Python MCP server  ── Remote Control HTTP ───────────►  ┐
  (~30 tools)        ── Python Remote Execution (UDP) ──►  │  Unreal Editor
                                                           │   ├─ UnrealAgentPlayer plugin
                                                           │   │   ├─ UAPAgentSubsystem (UFUNCTIONs)
                                                           │   │   ├─ log capture ring buffer
                                                           │   │   ├─ input injection (Slate, in-process)
                                                           │   │   ├─ fake IMotionController (VR pose)
                                                           │   │   └─ Blueprint helper discovery
                                                           │   ├─ Remote Control API (built-in)
                                                           │   └─ Python Editor Script Plugin (built-in)
                                                           ┘
  • Remote Control HTTP (127.0.0.1:30010) — structured calls to the subsystem's UFUNCTIONs. Deterministic, typed, per-port (so two editors are addressed independently).
  • Python Remote Execution (UDP multicast 239.0.0.1:6766) — arbitrary import unreal; ... scripting for anything not exposed as a UFUNCTION (load a level, read a pawn transform, start PIE).

Input injection is in-process via Slate — FSlateApplication::ProcessKeyDownEvent after focusing the PIE viewport — so it does not depend on the editor being the OS-foreground window, and two editors in separate processes can each be driven independently. See docs/architecture.md.


Features

FamilyToolsWhat it does
Statusbridge_statusLiveness of the editor, Remote Control, and Python Remote Exec; plugin version.
PIEpie_start pie_stop pie_pause pie_resume pie_statusDrive Play-In-Editor; query phase + elapsed time.
Inputinput_key input_mouse_move input_mouse_button input_axis input_gamepad input_sequenceInject keyboard / mouse / gamepad input into the running game (in-process, focus-independent).
VR inputinput_xr_button input_xr_pose input_xr_clearInject Quest Touch buttons and controller poses for headset-free VR testing.
Logslog_tail log_sinceCursor-based log capture with category + verbosity filtering.
Perfperf_stat perf_trace_start perf_trace_stopRead stat unit/fps (real draw/GPU/RT timings); Unreal Insights traces.
Perf baselinesperf_baseline_save perf_baseline_compareSave named perf snapshots; flag regressions over a tolerance.
Consoleexec_consoleRun any console command.
Pythonexec_pythonRun arbitrary import unreal editor scripting (level load, state reads, etc.).
Actorsactor_find actor_get_properties actor_set_propertyIntrospect and tweak actors in the world.
Read UIread_viewport_uiRead on-screen UMG text + focus so the agent can respond to prompts/menus instead of guessing.
Reportingreport_start report_assert report_note report_caption report_finishAuto-capture a run into a tabbed HTML report (verdict, checks, screenshot gallery, tool timeline, diagnostics) and auto-open it.
Screenshotsscreenshot_viewportCapture the viewport (HighResShot — see gaps).
Test helpershelper_list + CallTestHelperAuto-discover and call project-specific Blueprint/C++ assertion helpers (e.g. IsDoorOpen).
Editor menusui_menu_click ui_find_window ui_list_menusDrive native editor menu bar via Windows UIAutomation.
Editor focusFocusEditorWindow (UFUNCTION)Bring the editor window to front from in-process, for capture/native-chrome driving.

Full reference with parameters and return shapes: docs/capabilities.md.


Use cases

  • Closed-loop self-testing — the agent changes gameplay code, plays it, injects the input that exercises the change, and asserts on logs / actor state. See docs/use-cases.md.
  • Regression guarding — save a perf baseline on a known-good build; fail the loop if a later change busts the draw/GPU budget.
  • Headset-free VR iteration — exercise Quest Touch button flows in desktop PIE without donning a headset.
  • Reproducing and verifying bug fixes — script the exact input sequence that triggers a bug, confirm the fix, keep the script as a check.
  • Multi-editor workflows — drive two projects/editors at once; input routing is in-process per editor, addressed by Remote Control port.

Quickstart

:: 1. Install the plugin into your UE project (junction the Plugin/ folder)
mklink /J C:\Path\To\YourProject\Plugins\UnrealAgentPlayer C:\path\to\unreal-agent-player\Plugin

:: 2. Install the MCP server
cd unreal-agent-player\mcp-server
python -m venv .venv
.venv\Scripts\activate
pip install -e ".[dev]"
:: add the [windows] extra for editor-menu UIAutomation: pip install -e ".[dev,windows]"

Then enable Remote Control API, Python Editor Script Plugin, and Unreal Agent Player in your project, turn on Enable Remote Execution (Project Settings → Python), and register the MCP server with your client. Full steps: docs/setup.md and docs/claude-client-config.md.

Verify:

Call the bridge_status tool.

Expected: ue_running: true, rc_reachable: true, remote_exec_reachable: true, plugin_version: 0.0.1.


What's proven vs. known gaps

Proven (runtime-verified on a shipping VR project):

  • In-process key injection drives the PIE player pawn while the editor is not the foreground window, and works across two open editors.
  • read_viewport_ui reads live on-screen UMG text + focus (verified reading real connection/onboarding screens).
  • Real stat unit draw/GPU/render-thread timings; perf baseline save/compare.
  • PIE lifecycle, log capture, console + Python bridges, actor introspection, Blueprint helper discovery, editor-menu UIAutomation, in-process editor focus.
  • report_* produces a tabbed HTML run report with auto-captured timeline + screenshot gallery.

Known gaps / roadmap:

  • Screenshots omit UMG. screenshot_viewport uses HighResShot, which does not capture the UMG layer — so on-screen UI/prompts don't appear in the image. To read on-screen UI, use read_viewport_ui (deterministic text + focus). A backbuffer/desktop capture path for pixel-level UI (icon/glyph prompts with no text) is still on the roadmap.
  • VR HMD pose injection is deferred — no clean modular-feature hook; needs a fake IXRTrackingSystem.
  • VR controller pose-follow is limited in HMD-less PIE — a real XR runtime (e.g. Meta XR Simulator) wins modular-feature precedence over the agent's fake controller. Button injection is unaffected. See docs/architecture.md.
  • Windows only. See below.

Platform support

The plugin and Remote Control / Python bridges are cross-platform in principle, but this project is built and tested on Windows only. Editor-menu UIAutomation is Windows-specific (comtypes). macOS is unimplemented (no hardware to build/test the UE Meta fork; editor-menu driving would need an AppleScript/Accessibility rewrite).


Repository layout

Plugin/        UE editor plugin (C++)
mcp-server/    Python MCP server + tests
docs/          Setup, architecture, capabilities, use cases, writing test helpers
examples/      Example project-side test helpers

License

MIT. See LICENSE.

Ver o original no GitHub

Bom para

Relacionados