convez.ai
MCP ServerAutomaçãoAvançado
klemensms

MCP Computer Use

klemensms/mcp-computer-use

Controle semântico de apps macOS para agentes IA via screenshot e interação

Ver no GitHub
InicialPluginScore · de 0 a 100

Servidor MCP que permite ao Claude controlar aplicativos nativos do macOS de forma conversacional via Accessibility API e ScreenCaptureKit. Oferece captura de tela, cliques, digitação, atalhos e scroll em apps do allowlist, com segurança padrão e auditoria de todas as ações.

Instalação

$npx convez-ai klemensms/mcp-computer-use

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 4 meses
Versão
—
Issues abertas
0
Contribuidores
2

Casos de uso

  • Automação de tarefas em aplicativos desktop (Obsidian, Finder, Figma)
  • Assistência conversacional baseada em visão para apps nativos
  • Testes e RPA em interface nativa do macOS

Veredito

Ferramenta potente para automação de desktop no macOS com modelo de segurança bem pensado (allowlist, auditoria, redação). Beta estável, mas requer Xcode completo e setup inicial; ideal para quem precisa controlar apps nativos via agentes IA.

README

A tradução para o português deste README ainda está na fila.

mcp-computer-use

MCP server giving Claude Code (and any MCP client) semantic macOS app control — screenshot, click, type, keypress, scroll — via the Accessibility API and ScreenCaptureKit.

Status: v0.2.0-beta. macOS 14+. Full Xcode required for the postinstall Swift build (Xcode Command Line Tools alone are not sufficient — swift build ships with Xcode). Windows backend planned for v2.

Why

Existing browser-automation tools (Playwright, etc.) can't touch native apps. AppleScript is scripted, not conversational. This gives an agent the same kind of conversational, screenshot-driven control it has over a web page — but for Obsidian, Finder, Figma Desktop, or anything else you add to the allowlist.

Default-secure: per-app allowlist, rate limit, type_text secret scan, JSONL audit log, PROFILE presets. Agent can't act on apps you didn't bless.

Install

MCP config:

{
  "mcpServers": {
    "computer-use": {
      "command": "npx",
      "args": ["-y", "--package=@mcp-consultant-tools/computer-use@beta", "mcp-cu"],
      "env": {
        "MCP_CU_ALLOWED_APPS": "md.obsidian,com.apple.finder,dev.cmux.app,com.figma.Desktop,com.apple.calculator"
      }
    }
  }
}

First launch compiles the Swift helper from source via SwiftPM (requires full Xcode — install from the App Store; xcode-select -s /Applications/Xcode.app if it's already there but unselected). The helper imports BackgroundComputerUseKit and resolves on first build (~10s on a cold cache, ~1s warm). macOS then prompts for Accessibility + Screen Recording on ~/.mcp-computer-use/bridge — grant both.

Quick example

Agent to MCP: screenshot({app: "com.apple.calculator"}) → returns PNG + capture_id: "cap_abc12345"

Agent: click({x: 50, y: 180, capture_id: "cap_abc12345"}) → presses the 7 button in Calculator

Tools

ToolKindGated by
screenshotreadredaction for sensitive apps
list_windowsread—
get_frontmost_windowread—
check_permissionsread—
clickwriteallowlist + rate limit
type_textwriteallowlist + rate limit + secret scan
key_presswriteallowlist + rate limit
scrollwriteallowlist + rate limit
waitneutralrate limit

Set MCP_CU_READONLY=1 and only the reads are registered — write tools don't even appear in tool discovery.

CLI

Every MCP tool has a matching CLI command via mcp-cu-cli:

npx --package=@mcp-consultant-tools/computer-use mcp-cu-cli --help
npx --package=@mcp-consultant-tools/computer-use mcp-cu-cli screenshot --app com.apple.finder --out finder.png

Safety

See docs/technical/COMPUTER_USE_TECHNICAL.md for the full safety model. Short version:

  • Strict-AX (stealth) mode on by default
  • type_text rejects sk-…, ghp_…, AKIA…, xoxb-…, long base64
  • Every action logged to ~/.local/state/mcp-computer-use/audit.log (JSONL; text fields redacted)
  • PROFILE presets: stealth (default) / permissive / readonly / confirm

Acknowledgements

The Swift helper at native/macos/Sources/McpComputerUseHelper/ is a thin wrapper over BackgroundComputerUseKit (MIT © cam + anupam, dubdubdub labs), consumed via SwiftPM and pinned by commit SHA in native/macos/Package.swift. The wrapper translates upstream's Swift API into this project's existing JSON-over-stdio wire protocol; the TypeScript side and safety model are owned here. See NOTICE for full attribution and UPSTREAM_SYNC.md for the upgrade ritual.

Earlier releases (v0.1.x) shipped a vendored helper seeded from injaneity/pi-computer-use; that file was deleted in v0.2.0 — see the v0.2.0 release notes for the migration story.

Uses @mcp-consultant-tools/core for shared helpers across sibling MCP packages.

Roadmap

  • v1.1 — richer AX tools (axPressAtPoint, axFindTextInput), proper sensitive-app redaction pipeline, confirm-mode polish.
  • v2 — Windows backend (UIA + Windows.Graphics.Capture) slots in behind the existing NativeBridge interface.

Docs

License

MIT. See LICENSE for dual copyright attribution (Zane Chee upstream + Klemens Stelk this project).

Ver o original no GitHub

Bom para

Relacionados