convez.ai
MCP ServerAutomaçãoAvançado
88plug

Screen MCP

88plug/screen-mcp

Automação de desktop Linux Wayland com visão e controle para modelos de IA

Ver no GitHub
InicialPluginScore · de 0 a 100

Servidor MCP que dá a um modelo visão e controle sobre um desktop Linux Wayland/GNOME. Captura tela, clica, digita, scrolla e arrasta em qualquer aplicação, com suporte opcional a OCR e groundinging de ícones via OmniParser. Executa em CPU-only e é 100% Python.

Instalação

$npx convez-ai 88plug/screen-mcp

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
1
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 1 mês
Versão
—
Issues abertas
11
Contribuidores
1

Casos de uso

  • Automação de tarefas em aplicações desktop nativas
  • Agentes que precisam interagir com UIs reais além do navegador
  • Testes automatizados de interfaces gráficas em GNOME
  • Leitura e extração de informações de tela com OCR

Veredito

Ferramenta especializada e bem-executada para automação de desktop em ambientes Linux Wayland. Excelente para desenvolvedores que precisam de agentes com capacidades reais de UX, mas com escopo restrito ao GNOME/Wayland e alto overhead de dependências de sistema.

README

A tradução para o português deste README ainda está na fila.

screen-mcp

Give a model eyes and hands on a Linux Wayland desktop: screenshot, click, type, scroll, drag, and read any visible app.

plugin-validate License: FSL-1.1-ALv2 Claude Code plugin Docs

screen-mcp is an MCP server for Claude Code that lets a model see and operate your GNOME/Wayland desktop. It captures any monitor through PipeWire, drives the pointer and keyboard through the xdg-desktop-portal RemoteDesktop portal, and optionally reads the screen with OCR and grounds icons with an OmniParser ONNX model. It is for developers who want an agent that can use real desktop apps, not just a browser. It is pure Python and runs CPU-only.

Quickstart

Install the plugin in Claude Code:

/plugin marketplace add 88plug/screen-mcp
/plugin install screen-mcp@screen-mcp

Then run the one-time dependency setup (the server has system and Python deps the manifest cannot install for you):

# in the installed plugin dir (or a clone)
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt

On first use the desktop portal pops a consent dialog asking which monitor(s) to share. Pick one, and ask the model to take a screenshot:

Take a screenshot of my desktop and tell me which window is focused.

You should get back a labeled capture plus the focused-window name within a few seconds. The portal returns a restore token cached at ~/.config/mcp-screen so later runs are silent.

[!IMPORTANT] screen-mcp runs on Linux + Wayland + GNOME only, and grounding is CPU-only by design. See Requirements before installing.

What it does

The server turns one capture-and-act loop into MCP tools a model can call directly:

  • Screenshot any monitor or region, with numbered Set-of-Marks overlays and click coordinates.
  • Click, type, scroll, and drag in any visible app, including native Wayland apps that xdotool and XTEST cannot reach.
  • Read on-screen text with OCR (RapidOCR) and ground icons with an OmniParser ONNX model (both optional).
  • Sense changes: an ambient layer diffs frames so the agent knows when something opened or when an action was a no-op.
  • Cache learned screens: a write-through world model lets a recognized screen skip OCR on the next visit.
  • Gate destructive actions: an opt-in ack guard blocks close-combos and destructive-keyword clicks until the caller passes a confirmation token.

It also ships a drive-screen skill that encodes the locate → ground → act → confirm loop, so the model knows how to use the tools well out of the box.

MCP tools

These are the tools the server exposes. Every action also accepts space (view / desktop / norm, default view — coords as seen in the last screenshot), shot: true to return a screenshot after, verify: true to warn on no-change misclicks, force: true to bypass the user-takeover guard, and element: <id> to click an element id from the last annotated shot.

ToolWhat it does
screen_screenshotCapture the desktop. region / monitor to zoom, annotate=true for numbered marks, use_cache=true to reuse learned elements, fresh=true to force a current frame on a static monitor.
screen_list_monitorsMonitors (origin/size/scale), desktop bounds, focused windows.
screen_move_mouseMove the pointer to x,y.
screen_clickClick at x,y or in place. button, double.
screen_scrollWheel scroll by direction and amount.
screen_dragPress-drag from one point to another.
screen_keyPress a key or combo, e.g. Ctrl+L, Enter, Alt+Tab.
screen_typeType text into the focused window (Unicode via clipboard paste, ASCII via keysyms).
screen_focusRaise and give keyboard focus to a window by app/title/id.
screen_doRun a batch of ordered actions in one call.
screen_tourVisit several UI states, return a labeled thumbnail of each.
screen_read_pageAuto-scroll a scrollable view and accumulate every interactable.
screen_waitBlock until the screen settles, then optionally screenshot.
screen_sessionRecorder: start / stop / list / status / replay-path.
screen_reloadHot-reload the server in place after edits.
screen_diagHealth dump: session, cursor, grounding backends, world-model stats.

Requirements

[!IMPORTANT] screen-mcp targets a specific stack. It will not run elsewhere.

  • Linux + Wayland + GNOME. The awareness layer uses a bundled GNOME Shell extension; AT-SPI is the fallback for GTK apps.
  • Python 3.10+ (tested on 3.14).
  • GStreamer >= 1.28 (uses leaky-type; the older drop= was removed in 1.28).
  • PipeWire and xdg-desktop-portal-gnome.
  • wl-clipboard for the Unicode paste path in screen_type.
  • A DejaVu Sans Bold font for Set-of-Marks labels (falls back to PIL's default).

Grounding is CPU-only by design: the server hard-disables the GPU (CUDA_VISIBLE_DEVICES="") for predictable latency and no driver flake.

Install the system deps before the Python deps. See requirements.txt for the full pacman / apt one-liners.

# Arch / Manjaro
sudo pacman -S python-gobject gobject-introspection \
               gstreamer gst-plugins-base gst-plugins-good gst-libav \
               pipewire pipewire-pulse xdg-desktop-portal-gnome \
               wl-clipboard ttf-dejavu

# Python deps
pip install -r requirements.txt

The bundled window-info extension gives the awareness layer reliable focused-window and window-list data, and lets screen_focus activate windows directly. Installing it needs a one-time Wayland re-login.

gnome-shell-extension/window-info@local/install.sh
gnome-extensions enable window-info@local

For the kernel input backend, grant access to /dev/uinput by adding your user to the input group. The launcher (bin/screen-mcp) fails with a clear message if a required dep is missing, so a misconfigured install never silently half-works.

Manual MCP setup

If you are not using the plugin, wire the server in directly. Add to ~/.claude.json under mcpServers:

{
  "screen": {
    "command": "python3",
    "args": ["/path/to/screen-mcp/server.py"]
  }
}

Configuration

screen-mcp reads optional environment variables. A few common ones:

  • MCP_SCREEN_GUARD=1 — enable the ack gate. Destructive combos (Ctrl+W, Alt+F4), OCR-matched destructive keywords, and out-of-allowlist actions block unless the caller passes ack=<reason>.
  • MCP_SCREEN_APPS="firefox,terminal" — with the guard on, restrict actions to this allowlist of focused apps.
  • MCP_SCREEN_AMBIENT=0 — disable the ambient SENSE hint block.
  • MCP_SCREEN_CPU_THREADS=6 — ONNX intra-op thread count for OmniParser.
  • MCP_SCREEN_MAX_EDGE=2576 — screenshot downscale target (long edge).
VariableEffect
MCP_SCREEN_GUARD=1Enable the reliability ack gate (destructive combos / keywords / out-of-allowlist actions block until ack).
MCP_SCREEN_APPSWith guard on, allowlist of focused apps.
MCP_SCREEN_AUDIT_FRAMES=1Add pre/post frame hash + changed_bbox to each audit line (~100-500ms per action).
MCP_SCREEN_AMBIENT=0Disable the ambient SENSE hint block.
MCP_SCREEN_GUARD_PX=40Threshold for the user-takeover guard (live pointer vs last-commanded).
MCP_SCREEN_CPU_THREADS=6ONNX intra-op thread count for OmniParser.
MCP_SCREEN_MAX_EDGE=2576Screenshot downscale target (long edge).
MCP_SCREEN_NO_FRESH=1Disable forced fresh-frame capture on static monitors.
MCP_SCREEN_FOCUS_SETTLE_MS=150Delay after screen_focus activates a window.
MCP_SCREEN_NO_NUDGE=1Disable the pointer damage-nudge that primes a static monitor's frame.
PathWhat
~/.config/mcp-screen/tokenPortal restore token (one-time consent).
~/.local/share/mcp-screen/world/map.dbWorld-model cache (per-screen learned elements).
~/.local/share/mcp-screen/sessions/<sid>/Recorder trajectories, frames, replay.html.
~/.local/state/mcp-screen/actions.jsonlReliability audit log (one JSON line per action).
/tmp/screen_err.txtLast unhandled tool traceback (dev diagnostic).

Development

pytest -q          # tests run without a live D-Bus (conftest stubs)

Edit a .py, then call screen_reload in the running session to re-exec the server in place while preserving the MCP connection. On any tool exception the dispatcher writes the full traceback to /tmp/screen_err.txt; read it when debugging crashes.

Contributing

Issues and pull requests are welcome. Please run pytest -q before opening a PR. See CLAUDE.md for architecture notes and the hard-won ops details behind the capture and input layers.

License

FSL-1.1-ALv2 © 2026 88plug. The Functional Source License converts to Apache 2.0 two years after each release.

Ver o original no GitHub

Bom para

Relacionados