convez.ai
MCP ServerDevOps & InfraestruturaIntermediário
daviunx

regrun – Executor de Testes de Regressão Orientado por YAML

daviunx/regrun

Testes de regressão YAML para APIs, MCP, WebSocket e shell

Ver no GitHub
InicialPluginScore · de 0 a 100

MCP server que executa testes de regressão determinísticos contra APIs REST, servidores MCP, WebSockets e comandos shell, tudo definido em YAML. Suporta captura de variáveis (como JWTs), interpolação automática e compartilhamento de estado entre arquivos de teste sem necessidade de configuração de framework.

Instalação

$npx convez-ai daviunx/regrun

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 1 mês
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Validar regressões em APIs REST com tokens capturados
  • Testar servidores MCP com fluxos de setup e assertion
  • Executar testes de WebSocket e comandos shell em sequência ordenada
  • Compartilhar estado entre testes (JWT, IDs) sem wiring manual

Veredito

Ferramenta prática e bem pensada para testes de regressão determinísticos. Brilha na simplicidade YAML e no compartilhamento automático de variáveis, mas exige Python 3.12+ e uvx/fastmcp para suporte completo a MCP.

README

A tradução para o português deste README ainda está na fila.

regrun

Deterministic YAML-driven regression test runner for APIs, MCP servers, and WebSocket streams.

PyPI Python 3.12+ License: MIT


What is regrun?

regrun lets you define regression tests as YAML files and run them against live services — no test framework required. You describe what to call, what to assert, and what to capture; regrun handles execution, variable interpolation, and reporting. It supports four runners: REST APIs (httpx), MCP tools (fastmcp CLI), shell commands (bash), and WebSocket streams (websocket). Tests share a variable store across files, so a JWT captured in setup is available to every subsequent test without any wiring.


Installation

pip install regrun

Requires Python 3.12 or later.

The MCP runner requires uvx and the fastmcp CLI available on PATH:

pip install fastmcp

Quick Start

Create two test files for a fictional myapp running at http://localhost:8000.

tests/regression/00_setup.yaml — acquire a JWT:

meta:
  product: myapp
  layer: setup
  runner: httpx
  endpoint: "http://localhost:8000"

variables:
  RUN_ID: "{{timestamp}}"
  TEST_EMAIL: "regtest-{{RUN_ID}}@example.com"
  TEST_PASSWORD: "TestPass123!"

groups:
  - id: 1
    name: "Auth"
    priority: high
    tests:
      - id: "S.1"
        name: "Login and capture JWT"
        method: POST
        path: "/api/v1/auth/login"
        auth: none
        org_header: false
        body:
          email: "{{TEST_EMAIL}}"
          password: "{{TEST_PASSWORD}}"
        assert:
          status: 200
          json_path:
            "$.access_token": { exists: true }
        capture:
          APP_JWT: "$.access_token"

tests/regression/01_api.yaml — exercise the API with the captured token:

meta:
  product: myapp
  layer: api
  runner: httpx
  endpoint: "http://localhost:8000"
  default_auth: prod

auth:
  prod:
    type: bearer
    token: "{{APP_JWT}}"

groups:
  - id: 1
    name: "Items"
    priority: high
    tests:
      - id: "A.1"
        name: "List items returns array"
        method: GET
        path: "/api/v1/items"
        assert:
          status: 200
          json_path:
            "$": { not_empty: true }

Run the tests:

regrun run tests/regression/

Expected output:

tests/regression/  •  2 tests

  [PASS]  S.1  Login and capture JWT         (142ms)
  [PASS]  A.1  List items returns array       (38ms)

  2 passed, 0 failed  •  180ms

How It Works (Execution Model)

File ordering: The setup layer always runs first. All other files run alphabetically by filename. Numeric prefixes (00_, 01_, 02_) enforce the intended order.

Setup dependency: When you pass --layer api or --layer mcp, the setup file is auto-included and runs before the target layer. When setup runs as a dependency, --group and --priority filters are not applied to it — it always runs in full so captured variables stay available. Filters apply to setup only when it is the explicit target (--layer setup). Skip setup entirely with --skip-setup when variables are already populated from a prior run segment.

Variable persistence: File-level variables are merged once per file at parse time. A variable already set by an earlier file — for example RUN_ID defined in setup — is never overwritten by a later file's variables block. This ensures identifiers stay consistent across the entire run.

Layer concept: Tests are organised into four layers, processed in this order:

LayerPurpose
setupAuth, seed data, environment configuration
apiREST API surface tests
mcpMCP tool tests
chatWebSocket and streaming tests

CLI Reference

regrun run TEST_DIR [OPTIONS]

TEST_DIR is a path to a directory containing YAML test files.

FlagTypeDefaultDescription
--layersetup|api|mcp|chatallFilter to one layer (setup auto-included)
--group1,2,3allComma-separated group IDs
--priorityhigh|medium|lowallFilter groups by priority
--dry-runflagfalsePrint test plan without executing
--outputtext|jsontextOutput format
--verbose, -vflagfalseLog full request/response bodies
--fail-fastflagfalseStop on first failure
--skip-setupflagfalseSkip setup layer

Examples:

# Smoke test only
regrun run tests/regression/ --priority high

# MCP layer only
regrun run tests/regression/ --layer mcp

# Specific groups as JSON
regrun run tests/regression/ --group 1,3 --output json

# Preview without running
regrun run tests/regression/ --dry-run

YAML Schema Reference

meta block (required)

meta:
  product: myapp              # Used for reporting only — does not need to match any registered name
  layer: api                  # setup | api | mcp | chat
  runner: httpx               # httpx | fastmcp | bash | websocket
  endpoint: "http://localhost:8000"      # Base URL for httpx runner
  mcp_endpoint: "http://localhost:9000"  # MCP base URL — falls back to endpoint if omitted
  default_auth: prod          # Auth key applied to all tests without explicit auth:
  env_file: ".env.test"       # Path to .env file, relative to the test file's directory

The product field appears in report output. It does not need to match any external registry.

variables block

variables:
  RUN_ID: "{{timestamp}}"               # Unix timestamp + 4 hex chars (unique per run)
  TODAY: "{{date}}"                     # YYYY-MM-DD
  REQUEST_ID: "{{uuid}}"               # UUID4
  API_TOKEN: "{{env.MY_SECRET_TOKEN}}"  # Environment variable passthrough
  BASE_EMAIL: "admin@myapp.io"          # Static value

Built-in variables:

VariableDescription
{{timestamp}}Unix timestamp + 4 hex chars — unique per run, use as resource name suffix
{{date}}Current date as YYYY-MM-DD
{{uuid}}UUID4
{{env.VAR_NAME}}Reads VAR_NAME from the process environment

Full Jinja2 template syntax is supported. The engine runs in StrictUndefined mode: an undefined variable logs a warning and returns the raw template string rather than raising an exception.

Variables set by earlier files are preserved. Downstream files skip re-initialization of keys that already exist in the store.

auth block

auth:
  prod:
    type: bearer                # bearer | api_key
    token: "{{APP_JWT}}"
    org_header: "myapp"         # Sets X-Org-Slug header — omit if not needed
  service_key:
    type: api_key
    token: "{{SERVICE_API_KEY}}"

groups block

groups:
  - id: 1
    name: "Auth Flow"
    priority: high          # high | medium | low  (default: medium)
    context: prod           # prod | fresh | both  (default: prod)
    tests:
      - ...
  - id: 2
    name: "CRUD Operations"
    priority: medium
    tests:
      - ...

Test fields by runner

httpx (REST API)

- id: "A.2"
  name: "Create item"
  method: POST
  path: "/api/v1/items"
  auth: prod                  # Named auth key, "none", or omit to use default_auth
  org_header: true            # false to suppress X-Org-Slug
  body:
    name: "Widget {{RUN_ID}}"
    price: 9.99
  query_params:
    expand: metadata
  assert:
    status: 201
    json_path:
      "$.id": { exists: true }
      "$.name": { starts_with: "Widget" }
  capture:
    ITEM_ID: "$.id"

fastmcp (MCP tools)

- id: "M.1"
  name: "List items via MCP"
  tool: items_list
  args:
    status: "active"
    limit: 10
  auth: service_key
  assert:
    is_error: false
    json_path:
      "$[0].id": { exists: true }
      "$": { not_empty: true }
  capture:
    FIRST_ITEM_ID: "$[0].id"

bash (shell commands)

- id: "S.2"
  name: "Seed test user"
  runner: bash
  commands:
    - cmd: |
        docker exec myapp-postgres psql -U postgres -d myapp \
          -c "INSERT INTO users (email) VALUES ('seed@example.com') ON CONFLICT DO NOTHING;"
      capture:
        RAW_OUTPUT: stdout
  assert:
    last_exit_code: 0
    contains: "INSERT"

Bash commands run from the directory where you invoke regrun, not from the test file location. Use absolute paths or docker exec rather than relative paths.

websocket (streaming)

- id: "C.1"
  name: "Chat session produces response"
  url: "ws://localhost:8000/api/v1/ws/chat?session_id={{SESSION_ID}}"
  send:
    message: "What is the status of my account?"
    session_id: "{{SESSION_ID}}"
  wait_for: "agent_completed"      # Event type that terminates collection
  timeout: 60000                   # Milliseconds (overrides file-level timeout)
  ws_config:
    text_event: text_delta         # Override only if your server uses non-default field names
  assert:
    has_error: false
    json_path:
      "$.response_text": { not_empty: true }
      "$.event_count": { gt: 1 }
  capture:
    CHAT_RESPONSE: "$.response_text"

The runner connects, sends send as a JSON frame, collects events until wait_for is received, and returns an aggregated result dict:

FieldTypeDescription
response_textstrAll text_delta fragments joined
eventslist[str]Ordered list of all event types received
event_countintTotal number of events
tool_callslist[str]Tool names from tool_call events
duration_msfloatWall time from connect to termination event
errorstr|nullError message if an error event was received or timeout occurred

ws_config options (all have defaults — omit unless overriding):

FieldDefaultDescription
event_type_fieldevent_typePrimary key used to read the event type from each frame
event_type_fallbacktypeFallback key if primary is absent
text_eventtext_deltaEvent type whose payload contributes to response_text
text_fielddata.deltaDot-path to the text content within a text event
tool_call_eventtool_callEvent type that signals a tool was called
tool_name_fielddata.tool_nameDot-path to the tool name within a tool call event
error_eventerrorEvent type that signals an error
error_fielddata.contentDot-path to the error message within an error event

Per-test runner override — used in setup files that mix bash and httpx:

# In a file with meta.runner: bash, a single test can use httpx instead:
- id: "P.1"
  runner: httpx               # Overrides the file-level meta.runner
  method: POST
  path: "/api/v1/auth/login"
  auth: none
  org_header: false
  body:
    email: "{{TEST_EMAIL}}"
    password: "{{TEST_PASSWORD}}"
  assert:
    status: 200
  capture:
    APP_JWT: "$.access_token"

Pure api or mcp files should not use per-test runner: overrides — the file's meta.runner applies uniformly.


Assertion Vocabulary

Top-level assertions

KeyValuesRunner
status200 or [200, 201]httpx
is_errortrue|falsefastmcp
has_errortrue|falsewebsocket
last_exit_code0bash
containssubstring stringall runners

json_path operators

Each entry under json_path: maps a JSONPath expression to one operator:

OperatorExampleDescription
exists"$.id": { exists: true }Field presence check
equals"$.status": { equals: "active" }Exact match (string-coerced fallback)
contains"$.name": { contains: "Widget" }Substring
gt"$.total": { gt: 0 }Greater than
gte"$.count": { gte: 1 }Greater than or equal
lt"$.errors": { lt: 10 }Less than
lte"$.errors": { lte: 5 }Less than or equal
starts_with"$.key": { starts_with: "ntk_" }Prefix check
matches"$.slug": { matches: "^[a-z0-9-]+$" }Regex search
not_empty"$.items": { not_empty: true }Value is non-empty string, list, or dict

Note: numeric operators (gt, gte, lt, lte) are the correct names. greater_than, less_than, >=, and <= are not valid.


Variable Capture

capture:
  ITEM_ID: "$.id"                # JSONPath from JSON response
  OWNER_EMAIL: "$.owner.email"   # Nested path
  RAW_OUTPUT: stdout             # Full stdout (bash runner only)

Captured variables are stored in the shared VariableStore and are available to all subsequent tests in the run — including tests in later YAML files. This is how a JWT captured in 00_setup.yaml is accessible in 01_api_surface.yaml without any re-declaration.

Collision avoidance: suffix resource names with {{RUN_ID}} to prevent conflicts across runs:

body:
  name: "Test item {{RUN_ID}}"

Auth Patterns Guide

PatternYAMLWhen to use
File defaultmeta.default_auth: prodAll tests in file use the same auth
Per-test overrideauth: adminOne test needs different credentials
No authauth: noneLogin, register, org creation endpoints
Suppress org headerorg_header: falseBare-domain endpoints where X-Org-Slug causes 400 errors

auth: none is a string literal, not YAML null. Always write auth: none explicitly — writing auth: with no value parses as null and fails.

Multi-file auth flow: setup acquires credentials, downstream files consume them.

00_setup.yaml:

meta:
  runner: httpx
  endpoint: "http://localhost:8000"
# No default_auth — login endpoint needs no auth

groups:
  - id: 1
    tests:
      - id: "S.1"
        name: "Login"
        method: POST
        path: "/api/v1/auth/login"
        auth: none
        org_header: false
        body:
          email: "{{TEST_EMAIL}}"
          password: "{{TEST_PASSWORD}}"
        assert:
          status: 200
        capture:
          APP_JWT: "$.access_token"

01_api_surface.yaml:

meta:
  runner: httpx
  endpoint: "http://localhost:8000"
  default_auth: prod          # APP_JWT now available from setup

auth:
  prod:
    type: bearer
    token: "{{APP_JWT}}"      # Captured in 00_setup.yaml
    org_header: "myapp"

Complete Example

A self-contained two-file example for a fictional myapp REST service.

tests/regression/00_setup.yaml

meta:
  product: myapp
  layer: setup
  runner: bash
  endpoint: "http://localhost:8000"

variables:
  RUN_ID: "{{timestamp}}"
  TEST_EMAIL: "regtest-{{RUN_ID}}@example.com"
  TEST_PASSWORD: "TestPass123!"

groups:
  - id: 1
    name: "Seed"
    priority: high
    tests:
      - id: "S.1"
        name: "Verify database is ready"
        runner: bash
        commands:
          - cmd: "docker exec myapp-postgres pg_isready -U postgres"
            capture:
              RAW_OUTPUT: stdout
        assert:
          last_exit_code: 0
          contains: "accepting connections"

      - id: "S.2"
        name: "Login and capture JWT"
        runner: httpx
        method: POST
        path: "/api/v1/auth/login"
        auth: none
        org_header: false
        body:
          email: "{{TEST_EMAIL}}"
          password: "{{TEST_PASSWORD}}"
        assert:
          status: 200
          json_path:
            "$.access_token": { exists: true }
        capture:
          APP_JWT: "$.access_token"

      - id: "S.3"
        name: "Create API key"
        runner: httpx
        method: POST
        path: "/api/v1/api-keys"
        auth: session
        body:
          name: "regression-key-{{RUN_ID}}"
        assert:
          status: 201
          json_path:
            "$.key": { starts_with: "ak_" }
        capture:
          API_KEY: "$.key"

auth:
  session:
    type: bearer
    token: "{{APP_JWT}}"

tests/regression/01_api_surface.yaml

meta:
  product: myapp
  layer: api
  runner: httpx
  endpoint: "http://localhost:8000"
  default_auth: prod

auth:
  prod:
    type: bearer
    token: "{{APP_JWT}}"
    org_header: "myapp"

groups:
  - id: 1
    name: "Items CRUD"
    priority: high
    tests:
      - id: "A.1"
        name: "List items"
        method: GET
        path: "/api/v1/items"
        assert:
          status: 200
          json_path:
            "$": { not_empty: true }

      - id: "A.2"
        name: "Create item"
        method: POST
        path: "/api/v1/items"
        body:
          name: "Regression item {{RUN_ID}}"
          price: 19.99
        assert:
          status: 201
          json_path:
            "$.id": { exists: true }
            "$.name": { contains: "Regression item" }
        capture:
          ITEM_ID: "$.id"

      - id: "A.3"
        name: "Get item by ID"
        method: GET
        path: "/api/v1/items/{{ITEM_ID}}"
        assert:
          status: 200
          json_path:
            "$.id": { equals: "{{ITEM_ID}}" }
            "$.price": { equals: "19.99" }

      - id: "A.4"
        name: "Delete item"
        method: DELETE
        path: "/api/v1/items/{{ITEM_ID}}"
        assert:
          status: 204

Run it:

regrun run tests/regression/

Environment Variables

VariableDefaultDescription
REGRUN_TIMEOUT30Per-test HTTP timeout (seconds)
REGRUN_MCP_TIMEOUT60Per-test MCP call timeout (seconds)
REGRUN_WS_TIMEOUT30Per-test WebSocket timeout (seconds)
REGRUN_VERBOSEfalseLog full request/response bodies
REGRUN_API_ENDPOINT—Override meta.endpoint globally (for CI)
REGRUN_MCP_ENDPOINT—Override meta.mcp_endpoint globally (for CI)

CI Integration

In CI, services run as Docker containers with network aliases instead of *.localhost domains. Use the endpoint override variables to point regrun at the container aliases.

GitLab CI:

regression:
  stage: test
  services:
    - name: myapp-api:latest
      alias: api
    - name: myapp-mcp:latest
      alias: mcp
  variables:
    REGRUN_API_ENDPOINT: "http://api:8000"
    REGRUN_MCP_ENDPOINT: "http://mcp:9000"
  script:
    - pip install regrun
    - regrun run tests/regression/

GitHub Actions:

jobs:
  regression:
    runs-on: ubuntu-latest
    services:
      api:
        image: myapp-api:latest
        ports:
          - 8000:8000
    steps:
      - uses: actions/checkout@v4
      - run: pip install regrun
      - run: regrun run tests/regression/
        env:
          REGRUN_API_ENDPOINT: "http://localhost:8000"

The endpoint override applies to every test file in the run. YAML files keep their local *.localhost URLs for developer use; CI overrides them without any file changes.


File Structure

Recommended test directory layout:

tests/regression/
  00_setup.yaml          # Setup: auth, seed data, environment checks
  01_api_surface.yaml    # REST API surface tests
  02_mcp_surface.yaml    # MCP tool tests
  03_chat_surface.yaml   # WebSocket / streaming tests

Numeric prefixes control alphabetical sort order. The setup layer is always processed first regardless of filename, but 00_ makes the intent explicit and keeps directory listings readable.


Development

Install dependencies and run the test suite:

poetry install
poetry run pytest

Tests live at tests/integration/cli/ and cover CLI behaviour end-to-end.


License

MIT

Ver o original no GitHub

Bom para

Relacionados