convez.ai
MCP ServerAutomaçãoIntermediário
jhinzzz

ScreenForge

jhinzzz/ScreenForge

IA testa, você fica com o pytest auto-reparável para Web, Android e iOS

Ver no GitHub
BomPluginScore · de 0 a 100

Motor de automação de UI alimentado por IA que converte descrições em linguagem natural em scripts pytest executáveis e auto-reparáveis. Funciona como agente MCP, inspecionando interfaces, decidindo ações e gerando código que sobrevive a mudanças de UI — suporta Web, Android e iOS sem gravação manual.

Instalação

$npx convez-ai jhinzzz/ScreenForge

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
6
Forks
1
Acompanhando
0
Saúde
71%
Último commit
há 3 meses
Versão
v0.11.1
Issues abertas
0
Contribuidores
2

Casos de uso

  • Gerar testes automatizados de UI a partir de descrições em linguagem natural
  • Testar aplicações móveis (Android/iOS) e web com o mesmo agente
  • Manter suites de testes que se adaptam a mudanças de UI automaticamente
  • Integrar automação de testes em pipelines CI/CD com scripts pytest reutilizáveis

Veredito

ScreenForge é uma solução sólida para quem precisa de automação de testes cross-platform com geração de código real. A abordagem de agente decide/executa e auto-healing diferencia bem de tools de record-and-replay, mas exige LLM robusto e integração cuidadosa em workflows existentes.

README

A tradução para o português deste README ainda está na fila.

ScreenForge

CI Python 3.11+ License: MIT

中文 | English

Your AI agent runs the test. You keep the pytest file.

Describe the flow in plain language. Watch the agent click through a live site. Walk away with a self-healing pytest script that survives UI changes — and runs in CI. (Drives real iOS + Android devices too, not just Chrome.)

ScreenForge is an AI-driven UI automation engine that turns natural language into executable test scripts. Unlike record-and-replay tools, you don't perform the actions yourself — the AI does it for you.

ScreenForge Live Mirror — the generated pytest grows beside the live page as the agent runs

Why ScreenForge?

Playwright CodegenBrowser UseMidscene.jsScreenForge
Need to perform actions yourself?YesNoNoNo
Generates replayable test scripts?YesNoNoYes (pytest)
Self-healing when UI changes?NoNoNoYes
Works as AI Agent tool (MCP)?NoYesNoYes
Drives real iOS / Android devices?NoNoNoYes

Core architecture: Your AI Agent is the brain (understands requirements, makes decisions). ScreenForge is the hands (executes UI actions, generates code).

Quick Start

pip install screenforge

# See the magic without any API key:
screenforge --demo

# For real usage, set your LLM key:
export OPENAI_API_KEY=sk-...

# Inspect the current page (returns DOM tree for your Agent to analyze):
echo '{"operation":"inspect_ui","platform":"web"}' | screenforge --tool-stdin

# Execute a single action:
screenforge --action click --platform web --locator-type text --locator-value "Login"

How It Works

You (or your AI Agent)          ScreenForge
        │                            │
        ├──── "Test the login" ─────►│
        │                            ├── inspect_ui (get DOM tree)
        │◄── DOM tree ──────────────┤
        │                            │
        ├──── click #email ─────────►│
        ├──── input "user@..." ─────►│
        ├──── click "Sign In" ──────►│
        │                            │
        │◄── pytest script ─────────┤
        │◄── Allure report ─────────┤

Each step: inspect → decide → act → verify. The AI decides, ScreenForge executes.

Features

  • Keep a real pytest file: Every run emits a replayable pytest script (Allure-instrumented) you can drop straight into CI — not a black-box agent run.
  • Self-healing engine: When the UI changes and a locator breaks, the engine auto-repairs it with confidence scoring and AST validation, so your committed test survives.
  • The agent does the clicking: You (or your AI agent) describe intent; ScreenForge inspects the DOM, acts, and verifies — you never record by hand.
  • Real devices, not just browsers: Drives Android (uiautomator2) and iOS (wda) physical devices over the same protocol — the one thing pure-web agents (Playwright MCP, Browser Use) can't do.
  • L1/L2 semantic cache: Same page + same instruction = instant response, no LLM call needed
  • Visual fallback: When DOM can't locate elements (Canvas, games), VLM parses screenshots
  • MCP server: Any MCP-compatible Agent can drive ScreenForge natively
  • Structured output: JSON Lines events + report/runs/<id>/ artifacts for CI integration
  • Live Mirror playground: Watch the generated pytest code grow line-by-line beside a live screenshot as the test runs — screenforge --playground. See the Playground Guide

Agent Integration (Claude Code / Cursor / Codex)

ScreenForge exposes itself as a tool for AI Agents. The standard loop:

# 1. Get page structure (your Agent analyzes it)
echo '{"operation":"inspect_ui","platform":"web"}' | screenforge --tool-stdin

# 2. Your Agent decides what to do, sends precise actions
screenforge --action click --platform web --locator-type text --locator-value "Login"

# 3. Verify the result, repeat
echo '{"operation":"inspect_ui","platform":"web"}' | screenforge --tool-stdin

For batch operations, use workflows:

screenforge --workflow ./workflows/login.yaml --platform web --json

Or start the MCP server for native Agent integration:

screenforge --mcp-server

GitHub Actions

Add ScreenForge to your CI pipeline:

- uses: jhinzzz/ScreenForge@v1
  with:
    platform: web
    workflow: ./workflows/login.yaml
    openai-api-key: ${{ secrets.OPENAI_API_KEY }}

Results are auto-uploaded as Allure artifacts. See action.yml for all inputs.

See Agent Integration Guide for the complete protocol.

Installation (from source)

git clone https://github.com/jhinzzz/ScreenForge.git
cd ScreenForge
python -m venv .venv && source .venv/bin/activate
pip install -e ".[all]"

Platform-specific extras:

  • Web only: pip install -e . (default, includes Playwright)
  • Android: pip install -e ".[android]"
  • iOS: pip install -e ".[ios]"
  • ML/cache: pip install -e ".[ml]" (sentence-transformers for semantic cache)

Configuration

# Required: LLM API key (OpenAI-compatible endpoint)
export OPENAI_API_KEY=sk-...

# Optional: custom endpoint (defaults to api.openai.com)
export OPENAI_BASE_URL=https://api.openai.com/v1

# Optional: model (defaults to gpt-4o)
export MODEL_NAME=gpt-4o

Or create a .env file (copy from .env_template).

Badge

If ScreenForge generates tests for your project, add this badge to your README:

[![Tests by ScreenForge](https://img.shields.io/badge/tests%20by-ScreenForge-blue?logo=pytest)](https://github.com/jhinzzz/ScreenForge)

Tests by ScreenForge

Learn More

ResourceDescription
Mobile SetupAndroid & iOS device connection guide
MCP Setup (3 min)Connect to Claude Desktop / Cursor / Cline / Claude Code
Agent GuideIntegration protocol for AI Agents
Capability MatrixSupported platforms, actions, and locators
Architecture Deep-DiveBrain/hands split, semantic cache, self-heal AST gates, hygiene-as-feature
ExamplesReal committed workflows + the green pytest they generated
Playground GuideLive Mirror — watch code + screenshots grow as the test runs
Workflow ExamplesYAML workflow templates
CHANGELOGVersion history

Contributing

See CONTRIBUTING.md for guidelines. Issues and PRs welcome!

License

MIT

Ver o original no GitHub

Bom para

Relacionados