convez.ai
SkillAutomaçãoAvançado
mjenkinsx9

skill-kit

mjenkinsx9/skill-kit

Testagem e autoaperfeiçoamento rigoroso de Claude Agent Skills

Ver no GitHub
InicialPluginScore · de 0 a 100

Plugin para Claude Code que autentica, avalia e melhora autonomamente Skills de agentes Claude através de um harness com 21 verificações estáticas, testes comportamentais, medição de trigger empírica e loop de autoaperfeiçoamento. Traz evidências rigorosas em vez de suposições sobre qualidade de skills.

Instalação

$npx convez-ai mjenkinsx9/skill-kit

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 5 meses
Versão
v0.2.2
Issues abertas
1
Contribuidores
1

Casos de uso

  • Validar e debugar skills de agentes Claude antes de deploy
  • Medir acurácia de triggers e valor agregado de skills em produção
  • Automatizar iterações de melhoria de skills com feedback empírico
  • Garantir conformidade com best practices da Anthropic em agent skills

Veredito

Ferramenta sólida e bem-arquitetada para quem trabalha sério com Claude Agent Skills. Traz rigor científico (21 checks, testes comportamentais, loops de autoaperfeiçoamento) em vez de vibes — essencial para times que precisam medir e iterar skills em produção.

README

A tradução para o português deste README ainda está na fila.

🧰 skill-kit

The skill-testing workshop, packaged as a Claude Code plugin

Author, evaluate, security-review, and autonomously improve Claude Agent Skills — in any project, with evidence, not vibes.

CI Harness checks Bash Python Platform Made for Claude Code License: MIT


The core idea: a skill is only as good as the evidence behind it. skill-kit brings the full evaluation stack from the skill-testing dev-bench to any project as an installable plugin — a 21-check static harness, a live behavioral check, empirical trigger measurement, a blind value-add baseline, and an autoresearch loop that iteratively tightens a SKILL.md and keeps only the changes that score better.

Skills it touches aim to follow Anthropic's published agent-skills best practices (overview · Claude Code skills) — and the tooling exists to prove it rather than assume it.

✨ What's inside

LayerToolWhat it provesCost / cadence
🧹 Staticcheck-skillThe skill is well-formed — 21 checks: frontmatter, naming, trigger language, tree-wide secret & security-smell scans, sidecar hygieneFree · every edit
⚡ Behavioralbehavioral-checkThe documented commands actually run against the real system today~0 tokens · on touch
🎯 Empirical triggertrigger-accuracyThe skill really fires on its positive prompts and declines its negatives (observed, not self-predicted; Claude agent IDs or Pi/Codex transcript paths)Agent sessions · milestone
⚖️ Value-addvalue-add-testThe skill beats the cold model in a blind head-to-headLLM-judged · pre-promotion
🔄 Autoresearchimproving-skillsIterative modify → score → keep-or-revert loop that tightens an existing SKILL.mdHeavy · on demand
⚓ Goal anchoringgoal-new-skill · goal-improve-skillSkill work is driven by a measurable end state, not vibesFree · per skill

The harness is self-tested (10 pathological fixtures + 34 pytest tests) and CI enforces zero FAILs and zero WARNs on the shipped skills, on every push and PR.

🚀 Install

Inside Claude Code, on any machine:

/plugin marketplace add mjenkinsx9/mjenkins-toolbox
/plugin install skill-kit@mjenkins-toolbox

Update later with /plugin update skill-kit.

Because a plugin's bin/ joins the Bash PATH, every helper is callable by bare name (check-skill, score-skill, behavioral-check, value-add-test, improving-skills, goal-new-skill, goal-improve-skill, …) in any repo — no layout assumed. Run artifacts land in .skill-kit/runs/ in the consuming project (gitignore it).

Other harnesses

The Agent Skills (SKILL.md files) are the portable core, so skill-kit ships sibling manifests that point the same skills/ and commands/ directories at several agent CLIs. No content is moved or duplicated — each manifest just re-points the existing tree.

HarnessStatusManifestInstall / loadDocs
Claude Code✅ ported · validated (claude plugin validate . --strict).claude-plugin/plugin.json/plugin marketplace add mjenkinsx9/mjenkins-toolbox then /plugin install skill-kit@mjenkins-toolboxdocs
GitHub Copilot CLI✅ ported (skills + commands).plugin/plugin.jsoninstall the plugin per Copilot CLI; .plugin/plugin.json is first in its manifest search pathdocs
OpenAI Codex✅ ported · install-validated with Codex CLI.codex-plugin/plugin.jsoninstall via Codex plugins / .agents/plugins/marketplace.json catalogdocs
Cursor✅ ported (manifest conforms to schema; not locally validated).cursor-plugin/plugin.jsoninstall via Cursor pluginsdocs
Gemini CLI✅ ported — skills layer (commands/bin/ have caveats)gemini-extension.jsoninstall the extension; skills/ is auto-discovereddocs/gemini.md

Copilot CLI would read .claude-plugin/plugin.json as a fallback in its manifest search order (.plugin/plugin.json → plugin.json → .github/plugin/plugin.json → .claude-plugin/plugin.json), but Copilot defaults skills to skills/ while commands has no default — so the fallback alone would expose the skill but not the goal-* commands. To ship the full port, skill-kit adds a .plugin/plugin.json (first in that search order, ignored by every other harness) that declares both skills and commands. Codex and Cursor get a thin sibling manifest with metadata kept in sync with .claude-plugin/plugin.json, pointing at the unchanged skills/ (and, for Cursor, commands/). Only Claude Code has a validator available in this repo's toolchain, so those are conformance ports, not validator-verified. Gemini CLI auto-discovers a skills/<name>/SKILL.md directory, so its gemini-extension.json carries just the synced metadata and the improving-skills skill ports directly — with caveats: Gemini commands are TOML (the markdown goal-* commands don't auto-port) and it doesn't add bin/ to PATH. See docs/gemini.md.

🧭 Usage

# Lint a skill directory (or a SKILL.md directly) — exit 0 iff zero FAILs:
check-skill .claude/skills/my-skill

# Do the skill's documented commands still run against the real system?
behavioral-check .claude/skills/my-skill --dry-run

# Blind value-add baseline (preflight + tally; generation is agent-driven):
value-add-test .claude/skills/my-skill

# PATH shims for agent workflows/commands (print the exact protocol to load):
goal-new-skill goal.md
goal-improve-skill goal.md
improving-skills .claude/skills/my-skill/SKILL.md

# Empirical trigger smoke test with fresh Pi/Codex probe sessions:
trigger-accuracy run-probes .claude/skills/my-skill --runner pi --runs 1 --balanced --max-prompts 2
trigger-accuracy run-probes .claude/skills/my-skill --runner codex --runs 1 --balanced --max-prompts 2 --text-signal
/skill-kit:goal-new-skill summarizing-prs             # interview → goal.md for a not-yet-written skill
/skill-kit:goal-improve-skill .claude/skills/my-skill # measurable target for an improvement run
/skill-kit:improving-skills .claude/skills/my-skill/SKILL.md

The improving-skills loop scores each candidate against a fixed composite — mechanical floor (check-skill must pass every kept iteration), trigger accuracy (positive + negative fixture prompts), instruction quality (LLM-as-judge against the skill's own tests.md scenarios), and token efficiency (rewards shrinking, never penalizes a smaller body) — then runs the value-add baseline once on the final candidate, because a maxed composite proves a skill is tight, not that it beats just asking the model. See scoring.md and loop.md.

Note: the goal-* commands assume a /goal command in your environment — it is not part of this plugin or stock Claude Code. Without one, the generated goal.md still works as a manual acceptance checklist.

📁 Layout

.claude-plugin/plugin.json   Claude Code manifest
.plugin/plugin.json          GitHub Copilot CLI manifest → ./skills/ + ./commands/
.codex-plugin/plugin.json    OpenAI Codex manifest → ./skills/
.cursor-plugin/plugin.json   Cursor manifest → ./skills/ + ./commands/
gemini-extension.json        Gemini CLI extension (skills/ auto-discovered)
docs/gemini.md               Gemini CLI port notes + caveats
bin/                         deterministic helpers, auto-added to PATH
  check-skill                21-check static harness
  score-skill · token-count  mechanical scorers for the improvement loop
  behavioral-check           live tests.md command runner
  trigger-accuracy           empirical trigger helpers (Claude agent IDs or Pi/Codex transcripts)
  value-add-test             blind head-to-head scaffolding + tally
  goal-new-skill             PATH shim for /skill-kit:goal-new-skill
  goal-improve-skill         PATH shim for /skill-kit:goal-improve-skill
  improving-skills           PATH shim for /skill-kit:improving-skills
commands/                    /skill-kit:goal-new-skill · /skill-kit:goal-improve-skill
skills/improving-skills/     the autoresearch loop (/skill-kit:improving-skills)
reference/                   protocols the loop runs at milestones
tests/                       self-tests for the harness itself (fixtures + pytest)

🧪 Testing the tests

The harness itself is under test — because a linter you can't trust is worse than no linter:

bash tests/run-self-tests.sh    # 10 fixture skills with known verdicts
python -m pytest tests -q       # 34 tests for the Python eval scripts

🔁 Relationship to skill-testing

The skill-testing dev-bench is where the methodology is developed; this plugin is its distributable packaging. The shared tooling is co-developed as documented file pairs — see SYNC.md for the pairing table and the pre-release drift check, and MIGRATION.md for the extraction history.

💻 Requirements

Scripts are bash + a handful of POSIX tools + Python, and run identically on Linux, macOS, and Windows (Git Bash or WSL — the harness tries python3 / python / py -3 and skips the silent MS-Store shim). pip install pyyaml is the only Python dependency. All scripts use forward-slash paths and LF line endings (enforced via .gitattributes).

📄 License

MIT © 2026 Mike Jenkins

Ver o original no GitHub

Bom para

Relacionados