convez.ai
SkillProdutividadeIntermediário
Banner-Wang

Prompt Smith

Banner-Wang/prompt-smith

Iteração medida e científica para otimização de prompts

Ver no GitHub
InicialPluginScore · de 0 a 100

Skill para escrever, otimizar e testar prompts através de iteração automática e medida. Substitui a abordagem intuitiva por um loop experimental rigoroso, onde cada mudança é testada contra uma suite de avaliação fixa e mantida apenas se melhorar o score.

Instalação

$npx convez-ai Banner-Wang/prompt-smith

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
1
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Otimizar prompts existentes com base em métricas objetivas
  • Escrever prompts novos seguindo um processo estruturado de teste
  • Diagnosticar e corrigir regressões em suites de avaliação

Veredito

Ferramenta séria para quem trabalha com engenharia de prompts em produção. Força disciplina experimental em vez de iteração por intuição, reduzindo regressões silenciosas. Recomendado para times que precisam de prompts confiáveis e rastreáveis.

README

A tradução para o português deste README ainda está na fila.

prompt-smith

CI License: MIT

English · 简体中文

An Agent Skill for writing, optimizing, and testing prompts through automatic, measured iteration. Works with Claude Code, Codex, and Cursor — they share the same SKILL.md format.

Optimizing a prompt by eyeballing one output and rewriting on vibes is how prompts quietly regress. prompt-smith replaces that with a measured iteration loop: every change is a hypothesis tested against a fixed eval suite, and kept only if the score goes up. It is model-agnostic — the model that runs your prompt and the model that judges the output are both pluggable external commands.

              ┌─ author v0 (from scratch) ─┐
FRAME ────────┤                            ├─ BASELINE ─ [ DIAGNOSE → HYPOTHESIZE → EDIT → MEASURE → KEEP/REVERT ]* ─ CONVERGE
              └─ adopt v0 (existing prompt)┘                    └─────────────────────── repeat ───────────────────────┘

Why

Three failure modes kill prompt quality, and they all come from the same root — no measurement:

  1. No baseline. You can't tell whether an edit helped or hurt; you only know one output "looks nicer."
  2. Changing many things at once. Five edits in one pass — a score drop tells you nothing about which one caused it.
  3. Mistaking one example for the whole. You fix the case in front of you and silently break ten others (regression).

prompt-smith makes the eval suite the authority. You change one thing per iteration, re-run the same suite, and keep the change only if the aggregate score rises with no regressions. Prompt engineering becomes an experiment, not a craft you eyeball.

What it covers

One pipeline, two entry points:

  • Authoring — writing a prompt from scratch. Align on requirements → align on rules → turn both into the eval suite (the plan) → draft the shortest v0 that could pass it → harden it in the loop.
  • Iteration — improving a prompt that already exists. Adopt the existing version as v0 and go straight into the loop.

The only difference is where v0 comes from. From the baseline onward they're identical — which is why it's one skill, not two.

Install

A skill is just a directory with a SKILL.md. The three tools differ only in where they look for it — so installing is git clone into the right path. Pick global (available everywhere) or per-project (committed with one repo):

ToolGlobalPer-project
Claude Code~/.claude/skills/prompt-smith.claude/skills/prompt-smith
Codex~/.agents/skills/prompt-smith.agents/skills/prompt-smith
Cursor~/.cursor/skills/prompt-smith.cursor/skills/prompt-smith
# example: global install for Claude Code
git clone https://github.com/Banner-Wang/prompt-smith.git ~/.claude/skills/prompt-smith

The same checkout works for all three — nothing in the skill is tool-specific. To support more than one tool, clone into each path (or symlink one into the others):

git clone https://github.com/Banner-Wang/prompt-smith.git ~/.claude/skills/prompt-smith
ln -s ~/.claude/skills/prompt-smith ~/.agents/skills/prompt-smith
ln -s ~/.claude/skills/prompt-smith ~/.cursor/skills/prompt-smith

Each tool discovers the skill from SKILL.md automatically. Invoke it by asking in natural language — "write a prompt for…", "optimize this prompt", "reduce repetition in this prompt" — or directly with /prompt-smith if your client exposes skills as slash commands.

Requirements

  • Python 3.10+ — runs the eval engine (scripts/). CI covers 3.10–3.12.
  • PyYAML (optional) — only if you write the eval suite as .yaml. JSON suites work with no dependencies. pip install pyyaml
  • A way to call your models. The bundled runner/judge support any OpenAI-compatible HTTP endpoint (OpenAI, xAI, llama.cpp server, Azure-style, …) and the Claude CLI. Adding another provider is a few lines of shell — see references/runners.md.

No install step for the skill itself beyond cloning — the scripts are plain Python with no third-party imports (except optional PyYAML).

How it works

Each optimization run lives in a self-contained, inspectable directory:

<workdir>/
  config.env       # one place to declare both models (runner + judge); keys by env-var NAME, never values
  suite.yaml       # the eval suite (cases + asserts + judge rubric) — FROZEN once baselined
  runner.sh        # pluggable: turns a prompt into model output (reads config.env)
  judge.sh         # pluggable: scores output against a rubric (reads config.env; optional)
  prompts/
    v0.md, v1.md…  # one immutable file per version — never edited in place
  runs/
    v0.json…       # full eval result per version, each stamped with a suite_hash
  registry.jsonl   # machine-readable ledger: one line per version (score, hypothesis, decision, suite_hash)
  LOG.md           # human-readable narrative ledger (the "why")

Scoring: deterministic + subjective, measured separately

A prompt's quality has two kinds of property, scored with two rulers and blended by weight:

  • Hard constraints → deterministic asserts. Word/line caps, required/forbidden strings, JSON validity, regex. Objectively pass/fail — no model needed.
  • Soft qualities → LLM-as-judge rubric. Tone, coherence, persona fidelity, helpfulness. Things a rule can't capture, scored 0–1 by a (usually stronger) judge model.

Anything a deterministic check can decide never goes to the judge — and the judge owns only what asserts can't measure.

The fingerprint guard

The subtlest correctness rule: comparing two scores measured against different suites is meaningless. Every run is stamped with a suite_hash (over the cases, runner, judge, weights, and threshold). If you change the suite mid-stream, diff_runs.py refuses to compare runs from before and after — forcing you to re-baseline instead of drawing a false conclusion.

Quick start

# 1. Set up a workdir from the bundled templates
mkdir -p myprompt/prompts
cp ~/.claude/skills/prompt-smith/assets/suite.template.yaml   myprompt/suite.yaml
cp ~/.claude/skills/prompt-smith/assets/config.template.env   myprompt/config.env
cp ~/.claude/skills/prompt-smith/assets/runner.sh             myprompt/
cp ~/.claude/skills/prompt-smith/assets/judge.sh              myprompt/

# 2. Edit config.env — pick providers and models; keys are referenced by env-var NAME
#    (export the real key in your shell, e.g.  export XAI_API_KEY="sk-...")
$EDITOR myprompt/config.env

# 3. Write your starting prompt and your eval cases
$EDITOR myprompt/prompts/v0.md
$EDITOR myprompt/suite.yaml

# 4. Smoke-test the runner before trusting any score
echo '{"prompt":"Say exactly: hello world","case":"smoke","vars":{}}' | bash myprompt/runner.sh

# 5. Baseline
python3 ~/.claude/skills/prompt-smith/scripts/run_eval.py \
    myprompt/suite.yaml --prompt myprompt/prompts/v0.md --out myprompt/runs/v0.json

# 6. Iterate: edit a copy → measure → diff → keep or revert
python3 ~/.claude/skills/prompt-smith/scripts/run_eval.py \
    myprompt/suite.yaml --prompt myprompt/prompts/v1.md --out myprompt/runs/v1.json
python3 ~/.claude/skills/prompt-smith/scripts/diff_runs.py \
    myprompt/runs/v0.json myprompt/runs/v1.json

In practice you don't drive these by hand — you ask Claude Code to run the skill and it executes the loop, keeping registry.jsonl and LOG.md updated as it goes. The commands above are what it runs under the hood, and what you can reach for to inspect or re-run anything yourself.

Repository layout

SKILL.md                          # the skill itself — the loop Claude follows
scripts/
  run_eval.py                     # eval engine: render → run → assert + judge → score → report (exit code = pass/fail)
  diff_runs.py                    # per-case diff between two runs, with the suite_hash guard
  registry.py                     # structured ledger: record / log / best / show
references/
  authoring.md                    # writing v0 from scratch (start short, point don't re-describe, build in feedback)
  optimization-techniques.md      # the move set: which edit to try for which failure, and its side effect
  iteration-loop.md               # the loop in depth: noise, keep/revert edge cases, break-the-loop protocol
  test-cases.md                   # suite schema, every assert type, turning a vague quality into a testable case
  runners.md                      # the runner/judge contract + copy-paste provider implementations
assets/
  suite.template.yaml             # starting eval suite to copy
  config.template.env             # starting model config to copy
  runner.sh, judge.sh             # pluggable runner/judge, driven by config.env
tests/                            # pytest suite covering scoring, the fingerprint guard, and the ledger

Development

pip install pytest pyyaml
python3 -m pytest -v        # 11 tests, no network — uses a fake echo-runner

CI (GitHub Actions) runs the test suite plus compileall and bash -n syntax checks on every push.

License

MIT

Ver o original no GitHub

Bom para

Relacionados