Rock Tumbler — Otimizador Automático de Prompts
JaySOC-GIT/Rock-Tumbler-Prompt-Optimiser
Polir prompts automaticamente até a perfeição — deixe a IA fazer o trabalho tedioso
Ver no GitHubSistema multi-agente que automatiza o ciclo de engenharia, teste e refinamento de prompts usando a API Anthropic. Executa continuamente até atingir um platô de qualidade, teto de custo ou detecção de desvio semântico. Inclui detecção de retornos diminutos, comparação pareada de iterações e dashboard em tempo real.
Instalação
npx convez-ai JaySOC-GIT/Rock-Tumbler-Prompt-Optimiser1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Otimização sistemática de prompts em produção
- Engenharia de prompts com avaliação objetiva e rastreável
- Detecção automática de degradação semântica em refinamentos iterativos
Veredito

Ferramenta robusta para quem trabalha seriamente com engenharia de prompts em escala. Arquitetura técnica sólida com agentes especializados, dataset congelado para comparabilidade e detecção inteligente de desvio semântico. Exige profundidade técnica, mas resolve um problema real de produção.
README
A tradução para o português deste README ainda está na fila.
Rock Tumbler — Automated Prompt Optimisation System
"AI should do the tedious, mind-numbing work — so you can get back to the work that matters."
The Problem
When you generate a prompt, you have to evaluate it. In the most rudimentary form that means you write a prompt, run it, read the output, decide if it's good enough, tweak it, run it again. Repeat until satisfied.
That works. It also doesn't scale, it's inconsistent, and it's the kind of work that takes an hour when it should take minutes.
The Rock Tumbler automates that loop.
What It Does
The Rock Tumbler is a multi-agent prompt optimisation pipeline built on the Anthropic API. You give it a goal and a set of success criteria. It runs continuously — engineering prompts, testing them against your criteria, grading the outputs, and refining based on what failed — until it hits a quality plateau, reaches a cost ceiling, or you stop it.
Like a rock tumbler polishing stones: put in a rough prompt, let it run, stop it when you're happy with what came out.
Architecture
Three AI stages working in a continuous loop:
AI 1 — Prompt Engineer Generates and iteratively refines the prompt based on the original goal, grader feedback, and a map of previously identified dead ends. On the first iteration it generates from scratch. On every subsequent iteration it improves the current version based on what failed.
Frozen Evaluation Dataset Test cases are fixed and version controlled — the same inputs run every iteration. This is the methodological foundation: scores are only comparable if you're measuring against the same ruler every time. Users define their own success criteria directly in the widget; the system generates varied test inputs automatically.
AI 3 — Grader Two modes — general (prose and writing tasks) and code-specific (automatically detected from the goal). Uses structured outputs (schema-enforced via tool use) so evaluation results arrive as guaranteed Python dicts rather than text that needs cleaning and parsing.
Gate Check Layer Programmatic assertions run before the LLM grader on every output. Empty responses, refusals, length violations, and syntax errors are caught instantly in code — no tokens burned on obvious failures. Only outputs that pass basic checks go to the LLM grader.
Pairwise Comparison After each grading cycle, the current iteration's outputs are compared against the previous iteration's using a structured verdict call. Returns W/L/D (Win/Loss/Draw) with confidence level and reasoning. Displayed as labels on the score history chart — distinguishes genuine improvement from score noise.
Orchestrator Loop Manages all stop conditions, tracks semantic drift, handles backtracking, monitors ROI, and renders the live dashboard after every iteration.
Key Features
Semantic drift detection Uses a local sentence transformer model (all-MiniLM-L6-v2) to embed both the goal and each new prompt version. Euclidean distance between the two vectors measures how far the prompt has wandered semantically. If drift exceeds the threshold, the system backtracks to the last strong signal — like a metal detector operator retracing steps when the signal fades.
Diminishing returns detection Monitors score improvement across a rolling window. When gains fall below a threshold the system runs a confirmation window then stops — the NASCAR metric: at what point are you spending millions for fractions of a second?
ROI tracking Score improvement per dollar spent, tracked per iteration and displayed on the live dashboard.
Cost ceiling Hard stop when total API spend reaches the configured limit. Risk management built into the architecture.
Human-defined evaluation standard The user types their success criteria directly into the widget — one criterion per line. The system generates varied test inputs automatically. Human judgment defines what good looks like; the machine does the testing.
Parallel test execution All test cases within an iteration run simultaneously via ThreadPoolExecutor — significantly faster than sequential execution.
Live dashboard Six-panel matplotlib dashboard updating after every iteration: score history with W/L/D pairwise labels, semantic drift, ROI per iteration, cumulative cost vs ceiling, score trend line, and live status panel.
Stop Conditions
| Condition | Trigger |
|---|---|
DIMINISHING_RETURNS | Score improvement plateaus across N iterations |
COST_CEILING | Total spend reaches configured limit |
DIVERGENCE_LIMIT | Repeated semantic drift beyond recovery |
PERFECT_SCORE | 10/10 achieved |
USER_INTERRUPTED | Manual stop via button or kernel interrupt |
Design Philosophy
The Rock Tumbler was built from a specific frustration: prompt evaluation is genuinely important work that is also genuinely tedious. Writing a prompt and then sitting there manually reading outputs to decide if it's good enough is the kind of task that's hard to delegate and easy to do badly when you're tired of doing it.
The automation here isn't about replacing judgment — it's about moving judgment to where it matters. The user still defines what good looks like (the evaluation criteria). The system handles everything else: generating variation, running tests, measuring results, iterating, and stopping when further investment stops being worth it.
This reflects a broader design principle: AI should absorb the mechanical repetition so the human can focus on the decisions that actually require human judgment.
Enterprise and Assurance Relevance
For teams deploying AI systems in regulated or high-stakes environments, the question is rarely "can we make a better prompt?" It's "how do we know this system behaves within policy, and how do we prove it after every change?"
The Rock Tumbler's evaluation architecture speaks directly to that:
- Frozen evaluation dataset — a versioned, auditable set of test cases that defines acceptable behaviour. The same tests run every time. Any change in score is a real signal, not measurement noise.
- Gate check layer — deterministic, code-based assertions that catch policy violations before they reach the LLM grader. Auditable and explainable.
- Structured outputs — schema-enforced evaluation results. No ambiguous parsing, no silent failures.
- Full audit trail — every iteration logged with prompt version, score, cost, pairwise verdict, and grader narrative. The final report is a complete record of how the deployed prompt was produced.
- Cost controls — hard ceiling on autonomous spend. Operational risk management built in.
The evaluation harness is the foundation. The optimisation loop is the feature you add on top once measurement is trustworthy.
Tech Stack
| Component | Technology |
|---|---|
| LLM (all stages) | Anthropic Claude Sonnet via API |
| Embeddings | all-MiniLM-L6-v2 (local, sentence-transformers) |
| Evaluation | Frozen JSON dataset + programmatic gate checks + LLM-as-judge |
| Structured outputs | Anthropic tool use with JSON schema enforcement |
| Pairwise comparison | Claude structured verdict, temperature 0 |
| Parallel execution | Python ThreadPoolExecutor |
| Visualisation | Matplotlib (live inline Jupyter dashboard) |
| UI | ipywidgets (goal, criteria, cost ceiling, start/stop) |
| Environment | Python / Jupyter Notebook |
Setup
1. Clone the repo
git clone https://github.com/JaySOC-GIT/Rock-Tumbler-Prompt-Optimiser.git
cd Rock-Tumbler-Prompt-Optimiser
2. Install dependencies
pip install anthropic python-dotenv sentence-transformers numpy matplotlib ipywidgets
3. Add your API key
Create a .env file in the project directory:
ANTHROPIC_API_KEY=your-key-here
4. Open the notebook
Open Rock Tumbler Prompt Sharpening.ipynb in Jupyter and run all cells in order.
5. Run Enter your goal and success criteria in the widget, set a cost ceiling, and click Start Tumbling.
Notebook Cell Order
| Cell | Contents |
|---|---|
| 1 | %pip install dependencies |
| 2 | Core rock tumbler code (rock_tumbler_v3.py) |
| 3 | Gate check block — overrides grade() |
| 4 | Structured outputs block — overrides grader() and code_grader() |
| 5 | Pairwise comparison block — overrides render_chart() and rock_tumbler_jupyter() |
| 6 | Widget — goal input, criteria, cost ceiling, start/stop |
Built as part of the Anthropic API Certification Course
Built by someone who believes the best use of AI is giving you your time back.