convez.ai
MCP ServerAutomaçãoAvançado
MrSongP

Computer Use MCP Server

MrSongP/computer-use

Automação UI completa no Windows via MCP com screenshots e controle total de desktop

Ver no GitHub
InicialPluginScore · de 0 a 100

MCP server local para Windows que expõe UI Automation, screenshots, controle de entrada (mouse/teclado), lançamento de apps e gerenciamento de ciclo de vida. Funciona com Codex e Claude Code através de um runtime TypeScript e host nativo em C#.

Instalação

$npx convez-ai MrSongP/computer-use

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
2
Forks
0
Acompanhando
0
Saúde
28%
Último commit
há 4 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Automação de tarefas repetitivas em aplicações Windows
  • Teste automatizado de interfaces desktop
  • Captura de screenshots e análise de estado de janelas
  • Controle remoto de aplicativos via UI Automation

Veredito

Ferramenta sólida e completa para quem precisa automatizar fluxos desktop complexos no Windows. Requer setup técnico robusto (Node 20+, .NET 8+) mas oferece controle granular sobre UI Automation, screenshots estruturados e gerenciamento de ciclo de vida via MCP.

README

A tradução para o português deste README ainda está na fila.

computer_use

简体中文

computer_use is a local Windows computer-use plugin for Codex and Claude Code. It exposes window discovery, screenshots, UI Automation text trees, mouse and keyboard input, app launch, turn lifecycle, interrupt handling, and trace evidence through one TypeScript runtime and an MCP server. The plugin root is computer_use, with entrypoints in computer_use/.codex-plugin/plugin.json, computer_use/.mcp.json, and computer_use/skills/computer-use/SKILL.md.

Quick Start

Requirements:

  • Windows 10 or Windows 11
  • Node.js 20+
  • Codex CLI for Codex installs, or Claude Code CLI for Claude installs
  • .NET SDK 8+ for the C# native host build

Check the required command-line tools:

node --version
npm --version
dotnet --info
claude --version
codex --version

Install .NET SDK 8 with Windows Package Manager when dotnet --info does not list an SDK:

winget install --id Microsoft.DotNet.SDK.8 --exact --accept-source-agreements --accept-package-agreements

If dotnet is not available in the current terminal after installation, the installer also checks C:\Program Files\dotnet\dotnet.exe. If that file is missing, open a new terminal after installing the SDK or add C:\Program Files\dotnet to PATH.

Install into Codex from the repository root:

cd <path-to-cloned-repo>
npm run install:codex

Install into Claude Code:

cd <path-to-cloned-repo>
npm run install:claude

Install both:

npm run install:all

The npm installer installs TypeScript dependencies, builds the runtime, builds the C# native host, runs an MCP smoke test, registers the local marketplace, installs the plugin, and runs a Node-based doctor check.

Native Host Build

Build only the C# Windows native host:

npm run build:native

Build TypeScript plus C#:

npm run build:all

If you already built everything and only want to reinstall the current compiled artifacts:

npm run install:codex:compiled
npm run install:claude:compiled

The native-host builder uses dotnet build through .NET SDK 8+. It checks PATH, COMPUTER_USE_DOTNET_PATH, and the standard Windows install locations for dotnet.exe.

The .NET build targets net8.0-windows10.0.19041.0 so the native host can consume the Windows SDK C#/WinRT projections used by Windows.Graphics.Capture. Keep the TypeScript native-host launch path in sync with that target framework when changing the project file.

Capabilities

  • list_apps, list_windows, get_window, launch_app
  • get_window_state with direct MCP image content for screenshots and structured UIA nodes
  • click, click_element, press_key, type_text, scroll, set_value, drag, perform_secondary_action, activate_window
  • end_turn, turn lifecycle, physical Escape interrupt handling, and trace evidence
  • Codex adapter, Claude Code MCP adapter, and Windows native-host bridge

launch_app blocks duplicate cold launches by default. If an existing session is detected, the hook returns guidance to restore the app from windows.shell.taskbar; only an explicit force_new bypasses this behavior.

The Windows native host is turn-scoped. Normal completion, adapter close/shutdown, host stdio disconnect, process cleanup hooks, and a short native-host idle timeout release Computer Use resources; the host is restarted on demand because post-build startup is lightweight compared with desktop operations.

Development

cd <path-to-cloned-repo>
npm run typecheck
npm test

Focused checks:

npm --prefix computer_use run test -- tests/integration/codex-adapter.test.ts
npm --prefix computer_use run test -- tests/integration/claude-code-adapter.test.ts
npm --prefix computer_use run test -- tests/integration/stdio-runtime.test.ts
npm --prefix computer_use run test -- tests/integration/native-host-p5-smoke.test.ts
npm run doctor:codex
npm run doctor:claude

npm run codex:helper inside computer_use/ is only a local JSON-RPC helper harness for development. It is not the normal entrypoint after installing the Codex plugin.

Documentation

Windows Notes

  • Windows 10 and Windows 11 are both supported targets.
  • The native-host builder runs through .NET SDK 8+ and automatically checks the standard Windows dotnet.exe install path.
  • The taskbar target supports the primary Win10/Win11 taskbar and falls back to Shell_SecondaryTrayWnd for secondary-taskbar layouts.
  • Windows UI automation can affect the real desktop state, so keep target windows clear before running action tools.
Ver o original no GitHub

Bom para

Relacionados