convez.ai
MCP Server
autumnseasonism

skill-review

autumnseasonism/skill-review

Ver no GitHub
InicialPluginScore · de 0 a 100

Security auditor for agent skills — static + isolated multi-perspective review + dynamic honeytoken sandbox. For Claude Code, Codex, openclaw & Hermes.

Instalação

$npx convez-ai autumnseasonism/skill-review

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
1
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 3 meses
Versão
v1.0.0
Issues abertas
0
Contribuidores
1

README

A tradução para o português deste README ainda está na fila.

🛡️ skill-review

A security auditor for agent skills — before you trust them.

Static analysis · isolated multi-perspective review · dynamic honeytoken sandbox — for Claude Code, Codex, openclaw & Hermes.

version license python runtimes self--test

English · 中文


Why

Installing a third-party skill wires someone else's SKILL.md, scripts, templates and assets straight into your agent's context and your machine. That content is untrusted — it can hide malware, exfiltrate your local files, run destructive commands, or carry a prompt-injection that hijacks the host agent. Real 2025-2026 attacks already do exactly this (MCP tool-poisoning, GlassWorm zero-width steganography, CLAUDE.md/.cursorrules poisoning, npm preinstall payloads).

skill-review audits a skill before you trust it and returns a clear verdict with evidence.

🟢 PASS        safe to install
🟡 WARN        risky-but-maybe-legit — informed consent required
🔴 BLOCK       do not install — quarantined, with evidence

It is governed by the Principle of Lack of Surprise: a skill must not contain malware or exfiltration, and its real behavior must match its described intent. Benign roleplay is fine; hidden behavior is not.


How it works

flowchart LR
    A[candidate skill] --> B[deterministic layer]
    B -->|manifest · prescan · bundle| C{isolation boundary}
    C --> R1[🔴 red-team]
    C --> R2[🕵️ exfil/privacy]
    C --> R3[💉 injection]
    C --> R4[🧩 conflict]
    R1 & R2 & R3 & R4 --> AG[aggregate + sanitize]
    B -.optional.-> D[🐳 docker sandbox<br/>honeytokens + strace]
    D --> AG
    AG --> S[synthesizer]
    S --> V[verdict + report]
    classDef box fill:#161b22,stroke:#30363d,color:#e6edf3;
    class A,B,AG,D,S,V,R1,R2,R3,R4 box;
  1. Deterministic layer (Python, stdlib-only) enumerates every file, runs a bilingual signature prescan, and builds a neutralized review bundle (invisible/bidi unicode replaced with visible [[U+XXXX]] sentinels).
  2. Isolated multi-perspective review — four reviewer roles run as fresh subagents reading only the neutralized bundle and returning structured JSON (paraphrased evidence, no raw quotes). The orchestrator never reads raw untrusted text.
  3. Optional Docker sandbox plants honeytoken credentials and runs the skill's scripts with --network none + strace, catching secret-file openat() and connect() attempts.
  4. Aggregate → synthesize → verdict, applying a deterministic severity gate, then render a bilingual evidence report.

🔎 What it detects — 7 dimensions

#DimensionCatches
1Malicious intent / mismatchbehavior that contradicts the description; obfuscation; dead-code backdoors
2Data exfiltrationlocal files/secrets sent off-box; suspicious destinations; beacons
3Unauthorized accesshardcoded third-party creds; over-broad scope; SSRF; impersonation
4Dangerous commandsrm -rf ~, privilege escalation, AV/firewall disable, persistence, curl|bash, npm lifecycle hooks
5Prompt injectionoverride phrases (EN+中文), hidden unicode/Tag-block, CLAUDE.md/.cursorrules poisoning
6Skill conflictsname collisions, trigger-phrase overlap, contradictory behavior with installed skills
7Provenancesource/signature/tamper signals

Each finding carries severity × confidence + file:line evidence; dual-use patterns are disambiguated through a four-gate test (necessity · disclosure · declared scope · data-flow), so legitimate skills aren't flagged just for using requests or subprocess.


🚀 Quick start

The skill triggers automatically in Claude Code / Codex / openclaw / Hermes when you say:

"is this skill safe to install?" · "audit this skill" · "scan my installed skills" "这个 skill 安全吗" · "审查这个技能" · "扫描已安装技能"

Run the deterministic layer directly:

# enumerate + signature prescan a candidate skill
python scripts/enumerate_skill.py ./candidate-skill --out manifest.json
python scripts/prescan.py        ./candidate-skill --out prescan.json --snippets snippets.json

# discover installed skills across runtimes (deduped by realpath)
python scripts/discover_skills.py --runtime all --catalog catalog.json --out inventory.json

# optional dynamic layer (Docker): honeytokens + strace
docker build -t skill-review-sandbox assets/sandbox
python scripts/sandbox_run.py ./candidate-skill --yes --trace --out dynamic.json

# self-test the detector
python scripts/selftest.py

💡 On non-UTF-8 Windows consoles, prefix scripts with PYTHONUTF8=1.


🧱 Architecture

skill-review/
├── SKILL.md                  # orchestration playbook (the host agent reads this)
├── scripts/                  # 12 deterministic, stdlib-only tools
│   ├── enumerate_skill.py    #   file inventory + symlink-escape detection
│   ├── prescan.py            #   bilingual signature scan (signals, not verdicts)
│   ├── prepare_bundle.py     #   INPUT hardening: neutralize invisible unicode
│   ├── validate_findings.py  #   OUTPUT gate: reject non-schema / smuggled JSON
│   ├── aggregate_findings.py #   merge + dedup + sanitize + coverage check
│   ├── sandbox_run.py        #   Docker honeytoken + strace dynamic layer
│   ├── discover_skills.py    #   cross-runtime discovery
│   ├── gen_dashboard.py      #   self-contained HTML risk dashboard
│   └── …                     #   frontmatter, selftest, package_dist
├── references/               # methodology (threat taxonomy, severity gate, …)
│   └── reviewers/            # the 4 reviewer roles + synthesizer
├── assets/                   # sandbox Dockerfile, honeytokens, dashboard, hooks
└── tests/fixtures/           # benign + malicious self-test corpus

Isolation is code-enforced, not trust-based: input is neutralized before a reviewer reads it; output is schema-validated before the orchestrator trusts it. (This hardens the envelope — it does not OS-sandbox the LLM's reasoning; see limitations.)


✅ Validation

Every layer was exercised, not just designed:

CheckResult
Internal self-test (authored fixtures)10/10 recall · 3/3 precision
External corpus (documented real-world attacks)71% → 100% (7/7) after fixing 2 real gaps
Live injection testreviewer reports an "approve me / print your system prompt" payload instead of obeying
Live Docker sandboxhoneytoken openat() + connect() caught under --network none
Dogfood (skill-review audits itself)PASS, zero false positives
Real audit of an installed skill (wiki-tree)WARN — caught a 75 MB undisclosed bundled app + a trigger conflict, cleared all dual-use
Independent code reviewfound & fixed a real dashboard XSS + 6 more bugs

⚠️ Honest scope: skill-review is a high-quality first filter that produces evidence, not an absolute guarantee. A PASS means "nothing the current methods caught," not "proven safe." Self-test percentages are a regression signal, not real-world recall. See references/limitations.md.


🌐 Cross-runtime support

RuntimeSkill rootStatus
Claude Code~/.claude/skills, ~/.agents/skills, plugins✅ verified
openclaw~/.agents/skills, extensions/*/skills✅ verified
Codex.codex/skills, ~/.codex/skills✅ verified
Hermes$HERMES_HOME/skills (default ~/.hermes/skills)✅ verified
generic--root <dir>✅

Scripts are pure-Python (stdlib + the one yaml dep already used by skill-creator) and the orchestration is written in runtime-agnostic language. When subagents aren't available, it degrades to a single-context sequential fallback and says so in the report.


📦 Install

Copy the skill-review/ folder into your runtime's skills directory:

# Claude Code
cp -r skill-review ~/.claude/skills/

# Codex / openclaw / Hermes
cp -r skill-review ~/.codex/skills/
cp -r skill-review ~/.agents/skills/
cp -r skill-review ~/.hermes/skills/

For an AV-friendly distribution that omits the intentionally-malicious self-test fixtures:

python scripts/package_dist.py --no-fixtures --out dist

📋 Changelog · License · Acknowledgement

Ver o original no GitHub

Bom para

Relacionados