convez.ai
MCP ServerAutomaçãoAvançado
MikkoParkkola

AXTerminator

MikkoParkkola/axterminator

Automação de GUI macOS para agentes de IA com visão e controle nativo

Ver no GitHub
BomPluginScore · de 0 a 100

Servidor MCP que permite agentes de IA enxergar e controlar aplicações macOS via API de Acessibilidade. Oferece 34+ ferramentas para automação de GUI, captura de áudio, câmera e gerenciamento de desktops virtuais, construído em Rust com latência de 379μs por elemento.

Instalação

$npx convez-ai MikkoParkkola/axterminator

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
3
Forks
0
Acompanhando
0
Saúde
100%
Último commit
há 3 dias
Versão
v0.11.1
Issues abertas
3
Contribuidores
2

Casos de uso

  • Testes automatizados de GUI em background
  • Automação de fluxos de trabalho no macOS
  • Integração de agentes IA com aplicações desktop
  • Captura de áudio e câmera para processamento

Veredito

Ferramenta técnica robusta para automação profissional de macOS, mas restrita ao ecossistema Apple e exige conhecimento de APIs de acessibilidade. Ideal para equipes que precisam de GUI testing em larga escala ou integração profunda entre agentes IA e aplicações desktop.

README

A tradução para o português deste README ainda está na fila.

AXTerminator

CI Crates.io Downloads macOS Rust License Discussions Install in VS Code Install in Cursor

MCP server that gives AI agents the ability to see and control macOS applications.

Deploy · MCP Tools · CLI · Query Syntax · AX-first vs Vision-first · FAQ · Troubleshooting · Wiki · Known Limitations


Up to 34+ MCP tools (27 core + optional audio, camera, spaces). Background interaction via the macOS Accessibility API. 379us per element access. Audio capture, camera input, virtual desktop isolation. Your AI agent connects and your Mac becomes an extension of it.

Platform scope. AX-first-with-vision-fallback is macOS-only. iOS/iPadOS support is tracked in #43 as a future capability and, when shipped, will be screenshot + vision only — the idevice/RPPairing surface does not expose AX trees for third-party apps, and UIAutomation would require an on-device test runner (breaks the "mac-only agent, no on-device prereqs" contract). Reported screenshot latency via CoreDeviceProxy: ~350ms USB, ~700ms WiFi (credit @m13v). tvOS is unverified — the RSD surface is narrower than idevice's tvOS target suggests; please open an issue with device evidence if you've tested. See #42 for the AX-first-vs-vision-first positioning rationale.

Deploy

Tell your AI assistant (recommended):

Read https://github.com/MikkoParkkola/axterminator and install axterminator as my macOS GUI automation MCP server

Your agent will install the binary, wire itself up, and request accessibility permissions. Works in Claude Code, Cursor, Windsurf, and any AI with terminal access.

Or install manually:

brew install MikkoParkkola/tap/axterminator

Grant accessibility permissions: System Settings > Privacy & Security > Accessibility (add your terminal app).

axterminator mcp install auto-detects your AI client. Specify one with --client:

axterminator mcp install                       # Claude Desktop (default)
axterminator mcp install --client claude-code  # Claude Code
axterminator mcp install --client cursor       # Cursor
axterminator mcp install --client windsurf     # Windsurf
axterminator mcp install --client codex        # OpenAI Codex CLI
axterminator mcp install --client vscode       # VS Code Copilot
axterminator mcp install --client zed          # Zed
axterminator mcp install --dry-run             # Preview without writing

Also supported: gemini, amazon-q, lm-studio.

# From crates.io
cargo install axterminator --features cli

# Build from source
git clone https://github.com/MikkoParkkola/axterminator
cd axterminator
cargo build --release --features cli

# Manual JSON (Claude Desktop, Cursor, Windsurf, etc.)
# { "mcpServers": { "axterminator": { "command": "axterminator", "args": ["mcp", "serve"] } } }

For Codex (~/.codex/config.toml):

[mcp_servers.axterminator]
command = "axterminator"
args = ["mcp", "serve"]

Done. Your agent has 27 core tools (up to 34+ with all feature flags) to control any macOS app.

MCP Tools

CategoryToolsWhat the agent can do
GUIax_connect, ax_find, ax_click, ax_click_at, ax_type, ax_set_value, ax_get_value, ax_scroll, ax_drag, ax_key_pressConnect to apps, find elements, interact
Observeax_is_accessible, ax_screenshot, ax_get_tree, ax_get_attributes, ax_list_windows, ax_list_apps, ax_wait_idleCheck permissions, see UI state, screenshots
Verifyax_assert, ax_find_visual, ax_visual_diff, ax_a11y_auditAssert element state, AI vision fallback, visual regression, WCAG audit
Systemax_clipboard, ax_run_script, ax_undo, ax_session_info, ax_analyzeClipboard, AppleScript/JXA, undo actions, session state, UI analysis
Audioax_listen, ax_speak, ax_audio_voices, ax_audio_devicesCapture mic/system audio, text-to-speech, inspect installed macOS voices; optional Kokoro/Piper TTS via enhanced-tts
Cameraax_camera_capture, ax_gesture_detect, ax_gesture_listenCamera frames, gesture recognition
Spacesax_list_spaces, ax_create_space, ax_move_to_space, ax_switch_space, ax_destroy_spaceVirtual desktop isolation

Resources

Agents can browse app state without tool calls:

ResourceWhat
axterminator://appsRunning applications
axterminator://app/{name}/treeLive element hierarchy
axterminator://app/{name}/screenshotCurrent screenshot
axterminator://app/{name}/stateFocused element, window title
axterminator://system/displaysMonitor layout

Security

Destructive actions require confirmation via elicitation. HTTP transport requires bearer token auth. The AI has hands, not root.

CLI

axterminator apps                        # List accessible apps
axterminator find "Save" --app Safari    # Find element
axterminator click "Save" --app Safari   # Click it
axterminator screenshot --app Safari     # Capture screenshot
axterminator tree --app Finder           # Element hierarchy
axterminator mcp serve --http 8080 --token secret  # HTTP transport

Query Syntax

# Simple text — matches ANY of: title, description, value, label, identifier
axterminator find "Save" --app Safari

# By role
axterminator find "role:AXButton" --app Safari

# Combined role + attribute (AND)
axterminator find "role:AXButton title:Save" --app Safari

# By description (useful for apps like Calculator)
axterminator find "description:equals" --app Calculator

# By value
axterminator find "value:42" --app Calculator

# XPath-like
axterminator find "//AXButton[@AXTitle='Save']" --app Safari

How It Works

AXTerminator uses an undocumented behavior of Apple's Accessibility API: AXUIElementPerformAction() works on unfocused windows. Your agent clicks buttons in one app while you work in another. Neither notices.

379us per element access (Criterion, M1 MacBook Pro). Appium needs 500ms for the same thing.

7-strategy self-healing locators survive UI changes: data_testid, aria_label, identifier, title, xpath, position, visual_vlm.

AX-first vs Vision-first

OpenAI Codex computer use, Anthropic Claude Computer Use, Google Gemini Operator, and Perplexity Personal Computer all use the same paradigm: screenshot → vision LLM → pixel coordinates → cursor click. The vendor changes; the paradigm does not.

axterminator uses the opposite default: AX semantic tree → element reference → action. Vision is a fallback for the rare surfaces the AX tree cannot reach (canvas apps, games, OpenGL/Metal renderers).

DimensionVision-first (Codex CU / Claude CU / Gemini / …)AX-first (axterminator)
Speedseconds per action (vision round-trip + LLM)~379 µs per action (measured, Criterion)
Costvision tokens on every action~free (AX API call)
Reliabilitypixel-brittle — breaks on theme, font, layout changeselement-stable — semantic addressing survives UI changes
Backgroundvisible cursor movementtruly background — no visible cursor
Dense / labeled UIsstruggles with small targetsreads labels directly from the AX tree
Canvas / games / OpenGLworks (universal fallback)needs vision fallback (ax_find_visual)
Cross-platformanywhere a screenshot worksmacOS only (see #43 for iOS roadmap)

The key insight: axterminator is not a competitor to those agents — it is a layer underneath them. Any agent that can call an MCP tool (Claude Code, Codex CLI, Cursor, Windsurf, VS Code Copilot, Gemini CLI) can use axterminator as its hands. They provide reasoning; axterminator provides reliable, cheap, background-safe action.

Coverage gate. Before investing heavily in vision-fallback features, run the AX coverage audit below for one week on your actual app surface. If >95% of your actions resolve via AX, vision fallback is a nice-to-have. If <80%, expand the vision fallback path first. See benches/probes/README.md for the audit harness.

FAQ

Why use this instead of Codex computer use / Claude Computer Use / Gemini Operator?

You probably use both. Those agents call axterminator as an MCP tool. axterminator gives them AX-semantic actions that are ~1,000× faster and cost nothing per call. The vision model stays for tasks that genuinely need it (canvas apps, games), not for clicking a Save button in TextEdit.

Isn't vision-first simpler — no setup, works everywhere?

It works everywhere vision works: foreground, focused, slow, expensive. axterminator works in the background, costs nothing, and is sub-millisecond. For automation of native macOS apps — the majority of business software — AX is strictly better. Vision is the escape hatch, not the default.

The binding architecture decision is ADR-0001: AX-first With Vision Fallback.

What app surfaces does axterminator cover?

Any macOS app that exposes the Accessibility API: all native AppKit/SwiftUI apps, Electron apps, web apps in Chrome/Safari/Firefox, terminal apps. Canvas-only surfaces (Figma canvas, game renderers, video players) use ax_find_visual — the built-in vision fallback that tries AX first and falls back to VLM automatically.

Does it work with agents that are already doing computer use?

Yes. Add axterminator mcp install --client codex (or --client claude-code, etc.) and the agent gains 34 semantic tools it can call instead of pixel-clicking. It doesn't replace the agent's vision; it gives the agent a faster, cheaper path for the 90%+ of actions that don't need vision.

Who is it for beyond developers?

  • Executive assistants: file triage, calendar entry, copy-paste between apps — all in the background while you work
  • Sales ops: CRM entry, proposal generation across native apps, screen scraping structured data from dense UIs
  • Event coordinators: spreadsheet-to-calendar sync, venue research across multiple apps, confirmation email drafts — reliable because it reads labels semantically, not by pixel position

Known Limitations

OperationBackground?Notes
Click, press, read values, screenshotsYesCore operations work without focus
Text inputPartialSome apps need focused text field
Drag, system dialogsNoRequire cursor control / always grab focus
Gesture recognitionYesVerified: thumbs_up at 88.8% confidence
Speech transcriptionYesVerified: on-device, requires Dictation enabled

Platform coverage (see #43):

PlatformAX treeScreenshotApp launchStatus
macOS 12+✅✅✅Shipped; sub-ms AX path
iOS / iPadOS❌ (third-party apps)✅ (~350ms USB / ~700ms WiFi)✅Planned, screenshot + vision only — no AX semantics
tvOS❔❔❔Unverified — RSD surface narrower than idevice suggests
visionOS / watchOS❌❌❌Out of scope

Troubleshooting

SymptomCauseFix
Accessibility: DISABLEDNo permission grantedSystem Settings > Privacy & Security > Accessibility
Element not found for short labelsApp uses AXDescription not AXTitleTry description:label or inspect with axterminator tree
Application not foundWrong name or app not runningUse bundle ID: --bundle-id com.apple.calculator

AI agents: Fetch llms.txt for machine-readable installation instructions.

Feature Flags

The Homebrew formula includes all features. When building from source, select capabilities with feature flags:

cargo build --release --features "cli,audio,enhanced-tts,camera,spaces"
FlagWhat
cliCLI + MCP server (default)
audioMicrophone/system audio, speech
enhanced-ttsOptional Kokoro/Piper TTS engine routing and axterminator models tts downloads
cameraCamera capture, gesture detection
spacesVirtual desktop management
http-transportHTTP MCP transport with auth

Community

Acknowledgements

Inspired by Terminator by mediar-ai, which pioneered accessible desktop GUI automation on Windows.

For AI Agents

Machine-readable installation guide: llms.txt

Your agent can fetch this URL to get step-by-step installation, MCP config for every host, and troubleshooting.

Ecosystem

axterminator is part of a suite of MCP tools:

ToolDescription
mcp-gatewayUniversal MCP gateway — compact 12-15 tool surface replaces 100+ registrations
trvlAI travel agent — 36 MCP tools for flights, hotels, ground transport
nabWeb content extraction — fetch any URL with cookies + anti-bot bypass
axterminatormacOS GUI automation — 34 MCP tools via Accessibility API

License

AXTerminator is free for personal, research, educational, noncommercial open-source, and free public-good projects with attribution.

Business use requires a written commercial license. See LICENSE.md and COMMERCIAL.md.

Companies can buy a standard commercial-use license via GitHub Sponsors at EUR 500/month per named project.

Earlier releases published under MIT OR Apache-2.0 remain under those earlier license grants for those earlier versions only.

Ver o original no GitHub

Bom para

Relacionados