skill-testing
mjenkinsx9/skill-testing
Author, evaluate, security-review, and autonomously improve Claude agent skills - 21-check harness, behavioral/trigger/value-add evals, self-tested + CI
Instalação
npx convez-ai mjenkinsx9/skill-testing1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
README
A tradução para o português deste README ainda está na fila.
🛠️ skill-testing
A test harness and workshop for Claude agent skills
Author, evaluate, security-review, and autonomously improve skills — with evidence, not vibes.
Every skill in this repo aims to follow Anthropic's published agent-skills best practices (overview · Claude Code skills) — and the tooling here exists to prove it rather than assume it.
The core idea: a skill is only as good as the evidence behind it. Skills are built from real runs (never from imagination), linted by a 21-check static harness, exercised live against their own test scenarios, and — for generative skills — blind-judged against a no-skill baseline to confirm they actually beat just asking the model.
✨ What's inside
| Layer | Tool | What it proves | Cost / cadence |
|---|---|---|---|
| 🧹 Static | eval/check-skill.sh | The skill is well-formed — 21 checks: frontmatter, naming, trigger language, tree-wide secret & security-smell scans, sidecar hygiene, staging-drift | Free · every edit |
| ⚡ Behavioral | eval/behavioral-check.py | The documented commands actually run against the real system today | ~0 tokens · on touch |
| 🎯 Empirical trigger | eval/trigger-accuracy.py | The skill really fires on its positive prompts and declines its negatives (observed, not self-predicted) | Sub-agents · milestone |
| ⚖️ Value-add | eval/value-add-test.sh | The skill beats the cold model in a blind head-to-head | LLM-judged · pre-promotion |
| 🔄 Autoresearch | improving-skills | Iterative modify → score → keep-or-revert loop that tightens an existing SKILL.md | Heavy · on demand |
The harness is self-tested (10 pathological fixtures + 26 pytest tests) and CI enforces the repo invariants on every push and PR.
🚀 Quick start
git clone https://github.com/mjenkinsx9/skill-testing.git
cd skill-testing
pip install pyyaml # the only Python dependency
# Lint any skill directory (or a SKILL.md directly):
eval/check-skill.sh examples/reviewing-pull-requests
checking: examples/reviewing-pull-requests/SKILL.md
PASS 1. frontmatter parses as YAML
PASS 2. name 'reviewing-pull-requests' is valid
...
summary: 0 FAIL, 0 WARN
Exit code is 0 only with zero FAILs — WARNs are prompts to look, not blockers.
Note: open a Claude Code session with this repo as the working directory and everything in
.claude/skills/auto-loads — edits are picked up live, no restart needed.
📁 Repository layout
.claude/skills/ live skills, auto-loaded by Claude Code in this repo
improving-skills/ purpose-built autoresearch loop for SKILL.md improvement
staging/ WIP / under-review skills (NOT auto-loaded)
examples/ exemplary reference skills (always pass the harness cleanly)
templates/skill-skeleton/ scaffold: SKILL.md + tests.md + PROMOTION-CHECKLIST + sidecars
eval/ the evaluation stack (see table above)
tests/ self-tests for the harness itself (fixtures + pytest)
runs/ autoresearch working dirs (gitignored)
🔁 Workflows
1 · Author a new skill
Never write a skill from imagination. Do the task manually 2–3 times, then codify what actually worked — see CLAUDE.md for the full process and its anti-patterns.
# 1. After 2-3 real manual runs, scaffold a draft in staging/:
mkdir -p staging/<gerund-form-name>
cp -r templates/skill-skeleton/. staging/<gerund-form-name>/
# 2. Lint while drafting (WARNs are OK in staging/):
eval/check-skill.sh staging/<gerund-form-name>
# 3. Promote once it passes cleanly, tests.md captures real runs, and the
# PROMOTION-CHECKLIST.md is filled in:
cp -r staging/<gerund-form-name> .claude/skills/
eval/check-skill.sh .claude/skills/<gerund-form-name> # must be 0 FAIL
Optionally anchor the work first with /goal-new-skill <name> — it interviews you
into a verifiable end state and writes staging/<name>/goal.md.
For a generative/judgment skill, also run the value-add baseline before promoting — a maxed linter score proves structure, not value. See eval/value-add-test.md.
2 · Import an existing skill
Drop the folder under staging/<name>/, read it carefully, run the harness, and
promote with cp -r staging/<name> .claude/skills/ when satisfied.
3 · Security-review a skill
Same as import — but keep it in staging/ while reviewing so it can't auto-load
and trigger. The harness scans the whole skill tree (including scripts/ and
references/) for security smells (curl … | sh, rm -rf, base64 decodes,
outbound POSTs) and leaked credentials. Smells are WARNs — prompts to look, not
verdicts. The human reviewer decides.
4 · Autoresearch-improve a skill
/goal-improve-skill examples/reviewing-pull-requests # optional: set a measurable target
/improving-skills examples/reviewing-pull-requests/SKILL.md
Runs a modify → score → keep-or-revert loop against a fixed composite:
- Mechanical floor —
eval/check-skill.shmust pass every kept iteration - Trigger accuracy — positive + negative fixture prompts (simulated per iteration, measured empirically once at loop end)
- Instruction quality — LLM-as-judge against the skill's own
tests.mdscenarios - Token efficiency — rewards shrinking, never penalizes a smaller body
The composite proves a skill is well-formed and tight — not that it beats
just asking the model. So when the loop stops, it runs the value-add baseline once
on the final candidate and records the verdict. Output: a scratch branch with one
commit per kept iteration plus a runs/<run-id>/results.tsv audit log. See
scoring.md and
loop.md.
🔍 The eval harness in depth
The full 21-check table — levels, pragmas (allowlist secret,
allowlist windows-path), and interpretation guidance — lives in
eval/README.md. Highlights:
- Tree-wide security scanning — secrets and security smells are checked in
every
.mdand script file in the skill, not justSKILL.md - Trigger-language lint — descriptions must carry genuine "use when…" trigger phrasing with ≥2 concrete conditions; vague prose doesn't pass
- Sidecar hygiene —
tests.mdwith aLast verified:date is required outsidestaging/; reference docs belong inreferences/, scripts inscripts/, and orphan files thatSKILL.mdnever links get flagged - Staging-drift detection — a promoted skill is diffed against its
staging/twin so the two can't silently diverge
🧪 Testing the tests
The harness itself is under test — because a linter you can't trust is worse than no linter:
bash eval/tests/run-self-tests.sh # 10 fixture skills with known verdicts
python -m pytest eval/tests -q # 26 tests for the Python eval scripts
CI (harness.yml) runs on every push and PR:
examples/ must score 0 FAIL, 0 WARN, live and staging skills must score
0 FAIL, and both self-test suites must pass.
💻 Requirements
Scripts are bash + a handful of POSIX tools + Python, and run identically on
Linux, macOS, and Windows.
| Platform | Shell | Setup |
|---|---|---|
| Linux | system bash | python3 + pip install pyyaml |
| macOS | system bash/zsh (scripts use #!/usr/bin/env bash) | python3 + pip install pyyaml |
| Windows | Git Bash or WSL | python from python.org + pip install pyyaml — the harness tries python3 / python / py -3 and skips the silent MS-Store shim |
All scripts use forward-slash paths and LF line endings (enforced via
.gitattributes). Documented commands like cp -r are POSIX — run them in bash.
📚 Documentation map
| Doc | What it covers |
|---|---|
| CLAUDE.md | The build process, anti-patterns, promotion/demotion rules, sidecar policy |
| eval/README.md | The full 21-check table + all four eval layers |
| WRITING-DESCRIPTIONS.md | How to write the description: — the primary trigger mechanism |
| templates/skill-skeleton/ | The scaffold every new skill starts from |
🤝 Contributing
PRs welcome — see CONTRIBUTING.md for setup and the full guide. The bar is mechanical and enforced by CI:
eval/check-skill.shmust report zero FAILs on any skill you touch (zero WARNs if it's inexamples/).tests.mdscenarios must come from runs you actually performed — fabricated scenarios are the one unforgivable sin here.- If you change the harness, the self-tests must pass (and probably grow).
Security reports (harness bypasses, unintended execution) go through private vulnerability reporting — see SECURITY.md.
📄 License
MIT © 2026 Mike Jenkins