convez.ai
SkillAutomaçãoIntermediário
Kentobayashi

QA Framework para Claude Skills

Kentobayashi/claude-skills-QA-framework

Testa e valida Claude Skills com automação — gera relatório Excel estruturado

Ver no GitHub
InicialPluginScore · de 0 a 100

Framework Python automatizado que avalia a qualidade de Claude Skills testando precisão de triggers, qualidade de saída, falsos positivos e consistência. Gera relatórios Excel com scores em 4 dimensões críticas (Triggers, False Positives, Output Quality, Consistency).

Instalação

$npx convez-ai Kentobayashi/claude-skills-QA-framework

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
1
Acompanhando
0
Saúde
42%
Último commit
há 4 meses
Versão
—
Issues abertas
1
Contribuidores
2

Casos de uso

  • Validar qualidade de skills antes de liberar em produção
  • Medir consistência e falsos positivos de um skill
  • Gerar relatórios estruturados de QA para múltiplos skills
  • Avaliar critérios de saída customizados via LLM judge

Veredito

Framework sólido e bem documentado para QA estruturada de skills Claude. Ideal para equipes que precisam validar múltiplos skills com critérios claros e rastreáveis. Exige setup manual de test cases, mas compensa com automação de testes repetitivos via API.

README

A tradução para o português deste README ainda está na fila.

claude-skills-QA-framework

Automated quality assurance for Claude Skills — tests trigger accuracy, output quality, false positives, and consistency via the Anthropic API.


What is this?

A lightweight Python framework that automatically evaluates Claude Skills across four quality dimensions:

DimensionWhat it testsMax score
T — Trigger QualityDo the right prompts activate the skill?5 pts
FP — False PositivesDoes the skill stay silent when it shouldn't activate?5 pts
O — Output QualityDoes the output meet the defined quality criteria?5 pts
K — ConsistencyDoes the skill produce consistent results across runs?5 pts
Total20 pts

Thresholds:

  • ≥16/20 ✅ Released — ready for production
  • 11–15 ⚠️ Fix needed
  • ≤10 🔴 Rebuild from scratch

Results are exported as a formatted XLSX file with color-coded scores.


How it works

skill_eval_runner.py        ← Main runner — runs all tests via Anthropic API
skill_eval_data.py          ← Your skill definitions with test cases
skills/                     ← Your local skill library (SKILL.md files)

For each skill, the runner:

  1. Reads the SKILL.md from your local library
  2. Sends 3 positive trigger prompts → checks if the skill activates
  3. Sends 2 negative trigger prompts → checks for false positives
  4. Generates a full output using the main model
  5. Uses an LLM judge to score the output against your criteria
  6. Runs a second output generation to measure consistency
  7. Writes everything to a formatted Excel report

Setup

1. Clone this repo

git clone https://github.com/Kentobayashi/claude-skills-QA-framework.git
cd claude-skills-QA-framework

2. Install dependencies

pip install anthropic openpyxl pyyaml

3. Set your Anthropic API key

Windows CMD:

set ANTHROPIC_API_KEY=sk-ant-your-key-here

Windows PowerShell:

$env:ANTHROPIC_API_KEY="sk-ant-your-key-here"

macOS / Linux:

export ANTHROPIC_API_KEY=sk-ant-your-key-here

For a permanent setup on Windows, use:

[System.Environment]::SetEnvironmentVariable("ANTHROPIC_API_KEY", "sk-ant-your-key-here", "User")

4. Set your skill library path

Open skill_eval_runner.py and set REPO_PATH to your local skill library folder:

REPO_PATH = Path("./skills")   # Default: skills/ folder in this repo

Your skills should follow this structure:

skills/
├── core-skills/
│   └── my-skill/
│       └── SKILL.md
└── light-skills/
    └── another-skill/
        └── SKILL.md

5. Define your skills in skill_eval_data.py

The skill_eval_data.py file contains example entries. Replace them with your own skills:

SKILLS = [
    {
        "name": "my-skill",
        "category": "Core",               # Any label — used for filtering
        "folder": "core-skills/my-skill", # Path relative to REPO_PATH
        "trigger_pos": [
            "Prompt that should activate the skill",
            "Another trigger prompt",
            "A third trigger prompt",
        ],
        "trigger_neg": [
            "Prompt that should NOT activate the skill",
            "Another negative example",
        ],
        "output_criteria": (
            "Must contain: X, Y, Z. "
            "The output should do A and B. "
            "Vague or generic output scores low."
        ),
    },
]

Usage

# Test all skills
python skill_eval_runner.py

# Test one specific skill
python skill_eval_runner.py --skill my-skill

# Test by category
python skill_eval_runner.py --category Core

# Dry run — check structure without API calls (free)
python skill_eval_runner.py --dry-run

Cost & Time Estimates

ScopeAPI CallsEstimated CostEstimated Time
1 skill~9~$0.18~2–3 min
10 skills~90~$1.80~20–30 min
36 skills~324~$6.50~75 min

Costs depend on output length. The runner uses:

  • claude-haiku-4-5 for trigger detection and judging (cheap)
  • claude-sonnet-4-6 for output generation (quality)

You can change both models in skill_eval_runner.py under CONFIGURATION.


Output

Results are saved as a formatted .xlsx file:

skill-eval-20260510-1849.xlsx

The file contains:

  • Summary row (total skills, pass/fix/rebuild counts, average score)
  • Color-coded score columns per skill
  • Green/yellow/red gradient on the Total column
  • Notes with the LLM judge's reasoning

Folder Structure

claude-skills-QA-framework/
├── skill_eval_runner.py    # Main runner
├── skill_eval_data.py      # Skill definitions with test cases
├── skills/                 # Your skill library (add your SKILL.md files here)
│   └── (empty by default — add your own skills)
└── README.md

Tips for good test cases

trigger_pos — Make prompts realistic and varied:

  • Use different phrasings of the same intent
  • Include context that a real user would provide
  • At least one should be very specific, one more general

trigger_neg — Make them clearly unrelated:

  • Different domain (tech vs. business vs. creative)
  • Similar-sounding but different intent
  • Avoid edge cases that could genuinely be ambiguous

output_criteria — Be specific:

  • List concrete elements that must appear in the output
  • Mention what scores low ("generic advice without examples scores low")
  • Include framework names if the skill is based on a specific methodology

License

Apache-2.0 — see LICENSE for details.

Author

Kentobayashi


Built for the kaizen-consulting-skill project and the broader Claude Skills ecosystem.

Ver o original no GitHub

Bom para

Relacionados