MCP Computer Use
klemensms/mcp-computer-use
Controle semântico de apps macOS para agentes IA via screenshot e interação
Ver no GitHubServidor MCP que permite ao Claude controlar aplicativos nativos do macOS de forma conversacional via Accessibility API e ScreenCaptureKit. Oferece captura de tela, cliques, digitação, atalhos e scroll em apps do allowlist, com segurança padrão e auditoria de todas as ações.
Instalação
npx convez-ai klemensms/mcp-computer-use1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Automação de tarefas em aplicativos desktop (Obsidian, Finder, Figma)
- Assistência conversacional baseada em visão para apps nativos
- Testes e RPA em interface nativa do macOS
Veredito

Ferramenta potente para automação de desktop no macOS com modelo de segurança bem pensado (allowlist, auditoria, redação). Beta estável, mas requer Xcode completo e setup inicial; ideal para quem precisa controlar apps nativos via agentes IA.
README
A tradução para o português deste README ainda está na fila.
mcp-computer-use
MCP server giving Claude Code (and any MCP client) semantic macOS app control — screenshot, click, type, keypress, scroll — via the Accessibility API and ScreenCaptureKit.
Status: v0.2.0-beta. macOS 14+. Full Xcode required for the postinstall Swift build (Xcode Command Line Tools alone are not sufficient —
swift buildships with Xcode). Windows backend planned for v2.
Why
Existing browser-automation tools (Playwright, etc.) can't touch native apps. AppleScript is scripted, not conversational. This gives an agent the same kind of conversational, screenshot-driven control it has over a web page — but for Obsidian, Finder, Figma Desktop, or anything else you add to the allowlist.
Default-secure: per-app allowlist, rate limit, type_text secret scan, JSONL audit log, PROFILE presets. Agent can't act on apps you didn't bless.
Install
MCP config:
{
"mcpServers": {
"computer-use": {
"command": "npx",
"args": ["-y", "--package=@mcp-consultant-tools/computer-use@beta", "mcp-cu"],
"env": {
"MCP_CU_ALLOWED_APPS": "md.obsidian,com.apple.finder,dev.cmux.app,com.figma.Desktop,com.apple.calculator"
}
}
}
}
First launch compiles the Swift helper from source via SwiftPM (requires
full Xcode — install from the App Store; xcode-select -s /Applications/Xcode.app
if it's already there but unselected). The helper imports
BackgroundComputerUseKit
and resolves on first build (~10s on a cold cache, ~1s warm). macOS
then prompts for Accessibility + Screen Recording on
~/.mcp-computer-use/bridge — grant both.
Quick example
Agent to MCP:
screenshot({app: "com.apple.calculator"})→ returns PNG +capture_id: "cap_abc12345"Agent:
click({x: 50, y: 180, capture_id: "cap_abc12345"})→ presses the7button in Calculator
Tools
| Tool | Kind | Gated by |
|---|---|---|
screenshot | read | redaction for sensitive apps |
list_windows | read | — |
get_frontmost_window | read | — |
check_permissions | read | — |
click | write | allowlist + rate limit |
type_text | write | allowlist + rate limit + secret scan |
key_press | write | allowlist + rate limit |
scroll | write | allowlist + rate limit |
wait | neutral | rate limit |
Set MCP_CU_READONLY=1 and only the reads are registered — write tools don't even appear in tool discovery.
CLI
Every MCP tool has a matching CLI command via mcp-cu-cli:
npx --package=@mcp-consultant-tools/computer-use mcp-cu-cli --help
npx --package=@mcp-consultant-tools/computer-use mcp-cu-cli screenshot --app com.apple.finder --out finder.png
Safety
See docs/technical/COMPUTER_USE_TECHNICAL.md for the full safety model. Short version:
- Strict-AX (stealth) mode on by default
type_textrejectssk-…,ghp_…,AKIA…,xoxb-…, long base64- Every action logged to
~/.local/state/mcp-computer-use/audit.log(JSONL; text fields redacted) - PROFILE presets:
stealth(default) /permissive/readonly/confirm
Acknowledgements
The Swift helper at native/macos/Sources/McpComputerUseHelper/ is a
thin wrapper over BackgroundComputerUseKit
(MIT © cam + anupam, dubdubdub labs), consumed via SwiftPM and pinned by
commit SHA in native/macos/Package.swift. The wrapper translates upstream's
Swift API into this project's existing JSON-over-stdio wire protocol; the
TypeScript side and safety model are owned here. See NOTICE for full
attribution and UPSTREAM_SYNC.md for the upgrade ritual.
Earlier releases (v0.1.x) shipped a vendored helper seeded from
injaneity/pi-computer-use;
that file was deleted in v0.2.0 — see the v0.2.0 release notes for the
migration story.
Uses @mcp-consultant-tools/core for shared helpers across sibling MCP packages.
Roadmap
- v1.1 — richer AX tools (
axPressAtPoint,axFindTextInput), proper sensitive-app redaction pipeline, confirm-mode polish. - v2 — Windows backend (UIA +
Windows.Graphics.Capture) slots in behind the existingNativeBridgeinterface.
Docs
- User guide:
docs/documentation/computer-use.md - Technical reference:
docs/technical/COMPUTER_USE_TECHNICAL.md - Design spec:
docs/superpowers/specs/2026-04-22-mcp-computer-use-design.md - Release notes:
docs/release-notes/
License
MIT. See LICENSE for dual copyright attribution (Zane Chee upstream + Klemens Stelk this project).