qulib
TapeshN/qulib
Honest QA gap analysis for deployed web apps. Auth-aware scanning, real accessibility checks, and an MCP server for AI agents.
Instalação
npx convez-ai TapeshN/qulib1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
README
A tradução para o português deste README ainda está na fila.
Qulib
Release confidence for deployed web apps — the one question: should we ship?
Qulib fuses live-app quality evidence, automation maturity, and API coverage into a single scored verdict: ship / caution / hold / block. It prefers honest uncertainty over fake confidence: if auth blocks the crawl, coverage is thin, or data is incomplete, the report says so.
Design line: AI should explore unknown gaps; deterministic checks (crawl, axe, links, console) should scale. Cost Intelligence tracks LLM usage so repeated reasoning can graduate into checks you own in CI.
On npm: @qulib/core (engine + CLI qulib) and @qulib/mcp (MCP server for AI agents).
What Qulib does
- Crawls deployed apps (anonymous or authenticated via Playwright)
- Runs axe-core accessibility checks (WCAG 2 A/AA)
- Detects broken links, console errors, navigation failures
- Computes release confidence (0–100) with a coverage floor when too few pages were scanned
- Emits JSON and Markdown reports (or ephemeral JSON on stdout)
- Auth-aware: optional
detect-auth,explore-auth, form-login, and storage-state flows - Cost Intelligence (optional block on gap analysis): token usage, budget warnings vs per-call output ceiling, prompt fingerprints, maturity hints, conversion recommendations
Packages
| Package | Purpose |
|---|---|
@qulib/core | Analyzer engine and CLI (qulib) |
@qulib/mcp | MCP server exposing Qulib to AI clients |
Quick start (CLI)
Release confidence — the flagship command:
npx @qulib/core confidence --url https://example.com
Returns a verdict (ship / caution / hold / block) with a 0–100 score, top risks, and recommended next checks.
Add --repo to also score test-automation maturity and API coverage:
npx @qulib/core confidence --url https://example.com --repo .
Analyze (full gap report):
npx @qulib/core analyze --url https://example.com
Scaffold a test suite:
npx @qulib/core scaffold --url https://example.com --framework cypress-e2e
Score automation maturity (repo only, no URL needed):
npx @qulib/core score-automation --repo /path/to/repo
From a clone (repo root):
npm run analyze -w @qulib/core -- --url https://example.com
Smoke (no disk writes):
npm run smoke
Cost doctor (after a normal analyze that wrote output/report.json):
npx @qulib/core cost doctor
Quick start (MCP)
{
"mcpServers": {
"qulib": {
"command": "npx",
"args": ["-y", "@qulib/mcp"]
}
}
}
Ask your agent:
Use Qulib to analyze https://example.com and tell me if it's ready to ship.
Default analyze_app responses are summary-first (top gaps, cost summary, next deterministic checks). Pass includeFullReport: true for the full gapAnalysis including all scenarios.
CI integration (GitHub Actions)
Gate your deploys on Qulib's honest agent-summary verdict (pass / warn / fail) with a drop-in action. There are two surfaces — pick whichever fits your CI:
Option A — composite action (drop into an existing job)
# .github/workflows/qa.yml
jobs:
qa:
runs-on: ubuntu-latest
steps:
- name: Qulib analyze gate
uses: TapeshN/qulib/.github/actions/qulib-analyze@v1
with:
url: https://your-app.example.com
fail-on: fail # fail (default) | warn | never
qulib-version: 0.9.0 # pin a version for reproducible CI
Option B — reusable workflow (whole job in one line)
# .github/workflows/qa.yml
jobs:
qa:
uses: TapeshN/qulib/.github/workflows/qulib-analyze.yml@v1
with:
url: https://your-app.example.com
fail-on: warn
# secrets:
# auth-storage-state: ${{ secrets.QULIB_STORAGE_STATE }}
How the gate decides
The action runs @qulib/core analyze --url <url> --agent-summary, which emits the stable agent-summary JSON. The CLI itself always exits 0 — the verdict lives in the gate field — so the action reads gate and turns it into a CI exit code:
gate | fail-on: fail (default) | fail-on: warn | fail-on: never |
|---|---|---|---|
pass | ✅ pass | ✅ pass | ✅ pass |
warn | ✅ pass | ❌ fail | ✅ pass |
fail | ❌ fail | ❌ fail | ✅ pass |
A fail gate means a critical gap, a blocked scan, null/too-low confidence, or an auth-required surface that was never exercised — Qulib never silently passes an unevaluated deployment.
Every run uploads the agent-summary JSON as an artifact (qulib-agent-summary) and writes a job summary table with the gate, release confidence, top risks, and honesty notes.
Inputs (composite action)
| Input | Default | Description |
|---|---|---|
url | (required) | Base URL of the deployed app to analyze. |
fail-on | fail | Gate policy: fail, warn, or never. |
qulib-version | latest | npm version/dist-tag of @qulib/core run via npx. Pin for reproducibility. |
repo | "" | Optional path to the app repo for repo-aware analysis. |
config | "" | Path to a qulib config file relative to the working dir. |
auth-storage-state | "" | Path to a Playwright storage-state JSON for authenticated scans (write a secret to a file first — never inline it). |
extra-args | "" | Extra raw flags appended to qulib analyze (advanced). |
node-version | 20 | Node.js version to set up. |
install-browser | true | Install Playwright Chromium (Qulib crawls with Playwright). |
output-path | qulib-agent-summary.json | Where to write the raw agent-summary JSON. |
Outputs (composite action)
| Output | Description |
|---|---|
gate | The verdict: pass | warn | fail. |
release-confidence | Release confidence 0–100, or n/a when null. |
coverage-status | Coverage status enum (ok | thin | blocked-by-auth | …). |
blocked | true if the gate violated the fail-on policy (the job was failed). |
summary-path | Path to the written agent-summary JSON artifact. |
The composite action lives at
.github/actions/qulib-analyzeand the reusable workflow at.github/workflows/qulib-analyze.yml. Reference them at a stable tag (e.g.@v1) for reproducible CI.
Release confidence (short)
- Score starts from 100 and is reduced by high / medium / low gaps (see
gaps.ts). - If fewer than
minPagesForConfidencepages were scanned, confidence is capped at 40 and alow-coveragewarning is set—thin coverage must not read as “ready”. auth-requiredearly exit sets confidence 0 with no gap inventory: the deployment was not actually exercised.
Details: packages/core/README.md.
Confidence Layer (P3)
Qulib turns delivery signals into release confidence. The one question: ”Given everything we know right now, should we ship this?”
The Confidence Layer (v1, P3) fuses qulib's own evidence collectors into a single scored verdict:
| Verdict | Meaning |
|---|---|
| ship | Confidence ≥ 80, no blockers, all required sources evaluated |
| caution | Confidence 30–79, or an unknown signal on a required source |
| hold | Confidence < 30 |
| block | A hard-blocking evidence item, or nothing evaluable |
CLI:
qulib confidence --url https://example.com [--repo /path/to/repo] [--json]
MCP tool: qulib_score_confidence — composes analyze_app / qulib_score_automation / qulib_score_api into one fused verdict.
The 5 views (data model)
- Release Confidence — the fused score + verdict (available now; the
computeReleaseConfidenceoutput) - Delivery Traffic — time-series of verdicts per subject (schema +
diffConfidencehelper; persistence P4) - Inbox — human-judgment items: blockers, unknown signals, approvals needed (schema +
deriveInbox; queue P4) - Replay — provenance trace: how each score was computed and by which tool (
buildReplay) - Audit Trail — tamper-evident append-only ledger entry per verdict (
toAuditEntry; sink P4)
Honesty over fake confidence: sources that are not_applicable, unknown, or null are excluded from the denominator but reported in contributions and honestyNotes. An auth wall or empty corpus forces verdict=block — qulib never silently passes an unevaluated surface.
Agent decisions are one evidence source. The reserved agent-evidence kind in EvidenceSourceKindSchema lets external agentic decisions feed into the same aggregator in a future release without touching the math.
Documentation
- Core (CLI, API, Cost Intelligence)
- MCP server
- Source map — new contributors: start here to navigate the codebase
- Contributing
- Security
- Code of Conduct
- Manual testing checklist
Contributing
Issues and PRs are welcome. See CONTRIBUTING.md. Follow the Code of Conduct.
License
MIT — see LICENSE