convez.ai
MCP ServerAutomaçãoIntermediário
web-infra-dev

Midscene MCP Server

web-infra-dev/midscene

Automação UI multiplataforma via visão e linguagem natural

Ver no GitHub
Muito bomPluginScore · de 0 a 100

Servidor MCP que integra a plataforma Midscene de automação UI baseada em visão ao Claude. Permite automação de iOS, Android e web através de linguagem natural, sem dependência de seletores ou estrutura DOM.

Instalação

$npx convez-ai web-infra-dev/midscene

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
16 mil
Forks
1,2 mil
Acompanhando
70
Saúde
62%
Último commit
há 1 dia
Versão
v1.18.1
Issues abertas
44
Contribuidores
71

Casos de uso

  • Testes automatizados de UI em iOS e Android
  • Automação de fluxos web sem selectors
  • Validação visual de interfaces
  • Automação RPA multiplataforma com IA

Veredito

MCP robusto que expande capacidades do Claude para automação visual multiplataforma. Destacam-se a robustez contra mudanças de UI e suporte cross-platform (web, iOS, Android), ideais para QA e automação RPA.

README

A tradução para o português deste README ainda está na fila.

English | 简体中文

Official Website: https://midscenejs.com/

📣 Midscene Skills is here!

Use Midscene Skills to control any platform with OpenClaw

Showcases

💡 Why Midscene

Most UI automation — including AI tools that read the DOM or the accessibility tree — depends on page structure. That structure is fragile and incomplete: selectors break on every refactor, elements without semantic markup (icon-only buttons, custom controls, <canvas>) are invisible to it, native apps and cross-origin iframes are out of reach, and it cannot tell whether something actually looks right. Midscene works from the screenshot alone, and you describe each step in natural language:

  • Less maintenance — no selectors to chase when the UI changes.
  • Reach every element and surface — if a human can see it, Midscene can target it, even with no semantic annotations, on <canvas>, native apps, and cross-origin iframes.
  • Assert what users actually see — verify colors, highlights, layout, and rendered state, not just whether a DOM node exists.
  • Two ways to test — add Midscene to your Playwright / Vitest suite, or let an AI agent test autonomously via Skills and MCP.

Midscene is built for UI testing first, but the same vision-driven engine handles any UI automation task.

💡 What you can automate

Midscene works anywhere you can take a screenshot — web browsers, Android, iOS, HarmonyOS, desktop apps, and any custom interface — all through one API. Write automation with the JavaScript SDK or in YAML, hand it to AI agents via Skills and MCP, and look up every method (aiAct, aiQuery, aiAssert, and more) in the API reference.

🚀 Get started

✨ Driven by Multimodal Models

Midscene is all-in on pure vision for UI actions: element localization is based on screenshots only. It runs on multimodal models with strong UI localization, such as Qwen3.x, Doubao-Seed-2.0, GLM-4.6V, gemini-3.5-flash, and UI-TARS, including open-source options you can self-host. For data extraction and page understanding, you can still opt in to include DOM when needed.

Read more about Model Strategy.

📄 Resources

🤝 Community

🌟 Awesome Midscene

Community projects that extend Midscene.js capabilities:

📝 Credits

We would like to thank the following projects:

  • Rsbuild and Rslib for the build tool.
  • UI-TARS for the open-source agent model UI-TARS.
  • Qwen-VL for the open-source multimodal model Qwen-VL.
  • scrcpy and yume-chan allow us to control Android devices with browser.
  • appium-adb for the javascript bridge of adb.
  • appium-webdriveragent for the javascript operate XCTest。
  • YADB for the yadb tool which improves the performance of text input.
  • libnut-core for the cross-platform native keyboard and mouse control.
  • Puppeteer for browser automation and control.
  • Playwright for browser automation and control and testing.

📖 Citation

If you use Midscene.js in your research or project, please cite:

@software{Midscene.js,
  author = {Xiao Zhou, Tao Yu, YiBing Lin},
  title = {Midscene.js: Your AI Operator for Web, Android, iOS, Automation & Testing.},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/web-infra-dev/midscene}
}

✨ Star History

Star History Chart

📝 License

Midscene.js is MIT licensed.


Ver o original no GitHub

Bom para

Relacionados