convez.ai
MCP ServerAutomaçãoAvançado
Haruhiyuki

Vision-MCP

Haruhiyuki/vision-mcp

Automação desktop estruturada: menos tokens, mais rápido, reutilizável

Ver no GitHub
InicialPluginScore · de 0 a 100

Framework de interação com software desktop para agentes de IA que combina árvores de acessibilidade, OCR e visão para operar aplicações GUI com menor custo de tokens e execução mais rápida. Permite aos agentes mapear e reutilizar workflows em YAML, reduzindo exploração visual repetida.

Instalação

$npx convez-ai Haruhiyuki/vision-mcp

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
8
Forks
2
Acompanhando
0
Saúde
42%
Último commit
há 2 meses
Versão
@vision-mcp/cli@0.8.0
Issues abertas
0
Contribuidores
3

Casos de uso

  • Automação de tarefas repetitivas em aplicações desktop
  • Redução de custo de tokens em operações de UI com agentes
  • Mapeamento e reutilização de workflows GUI complexos
  • Integração de agentes com software legado sem APIs

Veredito

Solução técnica sólida para operação desktop de agentes, com arquitetura híbrida bem pensada (AX/UIA + OCR + visão). Produção-ready para equipes que precisam automatizar GUI complexas, mas demanda setup técnico considerável e conhecimento de YAML maps.

README

A tradução para o português deste README ainda está na fila.

Vision-MCP

English | 中文

Vision-MCP is a desktop software interaction framework for agents. It combines an MCP server, a reusable agent skill, and native desktop helpers so agents can operate GUI applications with lower token cost, faster execution, and less repeated visual exploration.

The framework uses a hybrid AX/UIA + OCR + vision-model architecture. It lets agents inspect software through the cheapest reliable signal first, then turn successful interaction paths into reusable vision-mcp.yaml maps made of actions and workflows.

Why It Exists

Agents can use screenshots and vision models to operate desktop software, but pure visual exploration is expensive and slow. Vision-MCP gives an agent a structured workflow:

  1. Explore a GUI task once with accessibility trees, OCR, screenshots, and visual fallback.
  2. Record stable states, locators, actions, postconditions, and workflows in a vision-mcp.yaml map.
  3. Reuse those actions and workflows on later runs.
  4. Patch the map when the UI shifts instead of rediscovering the whole task.

For repeated desktop workflows, this turns software use from one-off visual search into an increasingly reusable instruction layer.

How It Works

Reusable GUI Maps

On the first run, the agent explores the application state, clickable controls, state transitions, and expected results. Vision-MCP stores that knowledge in vision-mcp.yaml as:

  • reusable actions, such as clicking a specific control or entering text
  • higher-level workflows, composed from multiple actions
  • state anchors and postconditions used to verify progress
  • patch overlays that keep runtime fixes separate from trusted baseline maps

On later runs, the agent discovers available actions through MCP tools, reuses existing workflows when possible, and only falls back to exploration when the map does not yet cover the requested task.

Hybrid Exploration

Vision-MCP gives the agent multiple ways to understand a GUI:

  • native accessibility trees through macOS AX or Windows UIA
  • OCR for text regions and verification
  • screenshots and visual-model fallback for non-native or visually dense apps
  • window capsules for display, geometry, foregrounding, and live view support

Native structure is preferred when it is reliable. OCR and vision are used as fallbacks or verification layers.

Quick Start

Claude Code

Inside Claude Code:

/plugin marketplace add Haruhiyuki/vision-mcp
/plugin install vision-mcp@vision-mcp

The plugin installs the skill, MCP server configuration, examples, and helper bootstrap path.

Other MCP Hosts

For Codex, Cursor, Cline, OpenClaw, Hermes Agent, or any stdio MCP host, add a server like this:

{
  "mcpServers": {
    "vision-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "@vision-mcp/cli@latest",
        "serve",
        "--apps-root",
        "${HOME}/.vision-mcp/apps"
      ]
    }
  }
}

Then run:

npx -y @vision-mcp/cli@latest doctor
npx -y @vision-mcp/cli@latest init-apps

For host-specific configuration paths, macOS and Windows permissions, upgrade steps, and troubleshooting, see the Chinese install guide: INSTALL.md.

Core Capabilities

Platform Support

CapabilitymacOSWindows
Native helperSwift + ScreenCaptureKit + AX + Vision + IOKitPowerShell 5.1 + Win32 + UIA + System.Drawing + WinRT
Modern screenshotsSCScreenshotManager on macOS 14+PrintWindow PW_RENDERFULLCONTENT on Windows 8.1+
Accessibility treeAXUIElement + osascript fallbackUIA TreeWalker + MSAA fallback
OCRVision frameworkWindows.Media.Ocr
InputNSPasteboard paste + CGEventSendInput VK_PACKET with modifier support
ForegroundingNSWorkspace.activateSwitchToThisWindow, AttachThreadInput, and fallbacks
Health checkshealth.snapshothealth.snapshot with GDI/USER resource checks
Self-checkvision-mcp doctorvision-mcp doctor

Platform notes:

MCP Tool Surface

CategoryTools
Discoverylist_apps, list_workflows, describe, describe_workflow, describe_action, list_actions
Executionrun_workflow, perform_action
Low-level actionsclick_at, type_text, press_key, scroll
Exploration and visionsnapshot, annotated, OCR text click helpers
AX/UIAax-press for macOS AXPress and Windows UIA InvokePattern
Continuous correctionvision-mcp patch, patches
Window managementdisplays, capsule, restore, live-view
Diagnosticsdoctor [--watch sec]
Repairrepair_minimal --max-level 3

vision-mcp.yaml Map Model

Vision-MCP maps are designed to make GUI knowledge durable:

  • state: UI pages, menus, dialogs, modals, tooltips, and system modals
  • anchors: OCR, AX/UIA, visual hash, and window-title anchors with match policies
  • regions: shared sidebar, toolbar, keyboard, or app-specific UI zones
  • collections: repeated controls such as card grids, rows, or dialog buttons
  • multi-locators: priority chains such as accessibility, OCR text, nearby text, image patch, normalized bounding box, and vision fallback
  • workflows: multi-step procedures with inputs, timeouts, approval flags, and failure policy
  • patch overlays: runtime or agent-authored corrections with trust levels
  • parent state links: nested menus, dialogs, and modal chains

When authoring a map, follow the checklist in map-design.md. Full field details are in schema.md.

Agent Documentation

The agent-facing source of truth is the bundled skill:

Additional Chinese-language documentation:

Safety

Vision-MCP maps and workflows support explicit safety controls:

  • forbidden action categories such as payment, destructive actions, external communication, permission changes, and captcha handling
  • risk_level: requires_confirmation or destructive for actions that must go through approval
  • workflow-step approval_required: true
  • destructive workflow steps should use on_failure: abort
  • redaction patterns for passwords, credit cards, Steam Guard codes, bearer tokens, and similar secrets
  • action traces with before/after screenshots, locator hits, and postcondition results

Desktop-control tools inherit the permissions of the MCP host and native helper. Always respect application terms, user expectations, operating-system security prompts, DRM, anti-cheat systems, and elevation boundaries.

License

Apache-2.0. See LICENSE and third-party attribution in NOTICE.

Each source file includes an SPDX license header.

Disclaimer

Maps under examples/ describe public UI layouts of applications such as Apple Music, Notes, Activity Monitor, example ERP screens, and Steam on Windows. They exist to demonstrate Vision-MCP map structure and coverage patterns. They are not endorsed or authorized by the vendors of those applications. Trademarks and application copyrights belong to their respective owners.

Destructive workflow examples are included only to demonstrate safe map design patterns such as risk_level: destructive, approval_required: true, and on_failure: abort. They are not recommendations to perform destructive actions.

Ver o original no GitHub

Bom para

Relacionados