Canary Lab
ferterahadi/canary-lab
Loop de reparo automático para testes Playwright com agentes IA
Ver no GitHubMCP server que implementa um loop de reparo automático para testes Playwright. Captura falhas de teste, coleta evidências (logs, screenshots, traces, vídeos) e fornece contexto para agentes IA corrigirem o código e re-executarem os testes. Ideal para fluxos agent-first onde testes funcionam como especificação.
Instalação
npx convez-ai ferterahadi/canary-lab1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Automação de correção de falhas em testes e2e com IA
- Coleta estruturada de evidências e contexto de falhas de teste
- Workflow agent-first para desenvolvimento e QA contínuo
Veredito

Ferramenta especializada e bem pensada para times que usam testes como especificação e querem automação de reparo com agentes IA. Requer setup inicial e compreensão de Playwright, mas resolve um problema real em pipelines de desenvolvimento.
README
A tradução para o português deste README ainda está na fila.
Canary Lab
The AI repair loop for Playwright.
Canary Lab runs your local Playwright tests, captures the evidence around each failure, and hands your AI agent the context it needs to fix the app and rerun the check. A failed test leaves behind enough to understand what broke and continue from the same run, instead of digging through terminal scrollback. It is built for teams using tests as the spec.

Contents
- Repair Loop
- Quick Start
- Agent-First Workflow
- What Canary Lab Owns
- When to Use It
- Requirements
- Feature Folders
- Commands
- Repairing a Failed Run
- Limitations
- Documentation
- License
Repair Loop
- Canary Lab applies the selected envset and starts your local services.
- Playwright runs the feature tests.
- Logs, screenshots, traces, videos, summaries, and failure slices are saved under
logs/runs/<runId>/. - The AI agent reads the failure context, fixes the app or test, and signals
rerunorrestart. - Canary Lab continues from the same run until the check passes.
Quick Start
Create a Canary Lab workspace and open the local UI:
npx canary-lab init my-lab
cd my-lab
npm install
npm run install:browsers
npx canary-lab setup
npx canary-lab ui
This creates a workspace with sample features, installs Playwright browsers, registers AI agent tools, and opens the UI at http://localhost:7421. Use npx canary-lab ui --no-open to skip launching a browser.
After canary-lab setup, restart your AI agent so it can discover the Canary Lab tools. If they do not show up, refresh setup and start a fresh agent session with npx canary-lab setup --force.
Agent-First Workflow
Canary Lab is built for an agent-first workflow: an AI agent or other MCP client writes tests, starts or claims runs, reads failure context, makes fixes, and signals the next runner action. Canary Lab stays the local run monitor and source of truth for evidence; the agent is where diagnosis and code changes happen.
From your workspace, just ask:
/canary-lab run checkout locally, fix it if it fails, and run it again until it passes
Humans can also start runs from the UI, and custom clients can use the local HTTP API.
What Canary Lab Owns
Canary Lab has a narrow boundary. It does not define a new test language, assertion model, or browser runner — engineers and agents write normal Playwright tests, and Playwright still executes them. Canary Lab owns the run context around Playwright:
- Feature folder structure and scaffold conventions.
- Envset application and cleanup.
- Service startup, health checks, PTY streams, and shutdown.
- Run manifests, lifecycle events, logs, and retained artifacts.
- Failure slices, summaries, diagnosis journals, and agent handoff prompts.
- Rerun and restart signals after a fix.
Use plain Playwright when one command gives you enough context. Reach for Canary Lab when a failure depends on more than a browser assertion — which services were running, which env files were active, what the backend logged, and which artifacts were produced.
When to Use It
Good fit when you want to:
- Run e2e tests and have an AI agent fix the application code when they fail.
- Keep logs, screenshots, traces, videos, summaries, and diagnosis notes per run.
- Repair from saved run context instead of terminal scrollback.
- Start a frontend, API, worker, and dependent service together, with each service log attached to the run that used it.
- Run the same feature against
local,staging, orproductionenvsets without hand-editing.envfiles.
Probably unnecessary if a plain npx playwright test gives you enough context, you want self-healing locators, you do not need service orchestration or env switching, or you want a hosted dashboard that manages tests for you.
Requirements
- Node.js >= 20 and npm >= 9.
- A modern browser: Chrome, Firefox, or Safari.
- Local UI server on
http://localhost:7421; service orchestration throughnode-pty. - Optional repair agents: supported AI agent CLIs (
claude,codex) onPATH.
Feature Folders
A feature lives under features/<name>/ with feature.config.cjs, a Playwright config, specs under e2e/, and envsets under envsets/.
Create one from the UI or with:
npx canary-lab new feature checkout-discounts --description "Validate checkout discounts"
The UI's Add Test flow can also turn a PRD or uploaded document into a generated plan and Playwright files for review. Generated tests still run through Playwright.
Commands
npx canary-lab init <folder>
npx canary-lab setup
npx canary-lab ui
npx canary-lab mcp [--profile repair|verify|author|full]
npx canary-lab mcp doctor [--profile repair|verify|author|full]
npx canary-lab new feature <name> --description "..."
npx canary-lab env apply <feature> <set>
npx canary-lab env revert <feature>
npx canary-lab upgrade
uiis the primary human workflow.setuprefreshes the agent/tool registration described in Quick Start.mcpbridges local AI clients into the UI server, starting it if needed. It defaults torepair; use--profile verifyfor deployment checks,--profile authorfor authoring, or--profile fullfor the complete surface.new featureandenvare deterministic wrappers for scripts and agents.upgradesyncs scaffolded docs and skills in an existing project (not a dependency upgrade).
Repairing a Failed Run
When a run fails, Canary Lab pauses it, waits for a fix, and reruns from the same run — every fix ends in a rerun or restart signal. Two modes drive the fix:
- External heal (default) — an external MCP client claims the run, reads the saved context, and signals the fix over MCP.
- Auto-heal — Canary Lab spawns a local
claude/codexCLI in a PTY tab (select Claude or Codex in Settings).
See the guide for the full loop, MCP tool flow, and signal files.
Limitations
- Repairs depend on useful service logs.
- Envset runs overwrite target files while active. If the process is killed during backup or restore, reopen the UI and use the envset controls to recover.
- Envset values are not validated. Stale config can cause unclear test failures.
- The Linux and Windows workflows are not polished yet.
Documentation
- Guide — environment switching, run-output layout, repairing a failed run, evaluation reports, and external authoring.
- Contributing — code orientation, run architecture, and the build/test workflow.
- Changelog — release notes.