AutoResearch
Evarodenas/autoresearch
Loops de otimização autônoma com ratchet e git para code, prompts e configs
Ver no GitHubSkill para otimização autônoma e iterativa de qualquer artefato (código, prompts, conteúdo, modelos) usando o padrão de Karpathy. Define métricas congeladas em program.md e deixa um agente IA executar experimentos com git como histórico — cada commit mantido é uma melhoria genuína.
Instalação
npx convez-ai Evarodenas/autoresearch1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Otimizar prompts de IA iterativamente com feedback automático
- Melhorar código mediante experimentos quantificados e versionados
- Iterar em conteúdo ou configurações com métrica congelada
- Pesquisa de hiperparâmetros ou ajuste de modelos sem intervenção manual
Veredito

Implementação sólida e bem-arquitetada do padrão autoresearch de Karpathy. Segue boas práticas (métrica congelada, ratchet via git, histórico limpo) e oferece flexibilidade como skill Claude e CLI standalone. Útil para quem quer pesquisa/otimização sistemática e auditável.
README
A tradução para o português deste README ainda está na fila.
autoresearch
Autonomous, ratcheting optimization loops for any artifact — an open implementation of Andrej Karpathy's "autoresearch" pattern.
You write a program.md describing what "better" means. An AI agent then runs
experiments on your behalf: form a hypothesis → make one change → score it
against a frozen metric → keep it if it improved (git commit) or throw it
away if it didn't (git reset --hard). Every surviving commit is a genuine
improvement. The git history is the research record.
Works on code, prompts, content, models, and configs — anything you can attach a number to.
┌──────────────┐ writes ┌──────────────┐ reads & ┌──────────────┐
│ HUMAN │ ─────────▶ │ program.md │ executes ▶ │ AI AGENT │
└──────────────┘ └──────────────┘ └──────┬───────┘
│ one change
keep ◀── better? ── run frozen eval ◀── git commit ◀──────────┘
revert ◀ worse/crash ── git reset --hard
Why this design works
| File | Role | Who edits it |
|---|---|---|
program.md | The research spec: goal, metric, constraints, search space, budget | You |
| target artifact | The thing being optimized | The agent |
evaluate.* | How success is scored | Frozen — nobody touches it mid-run |
results.tsv | Append-only experiment log | The agent appends |
The wall between the frozen evaluation and the mutable target is the whole trick. The agent can't cheat by editing how it's graded, so it's safe to let it run unattended. Git enforces a monotonic ratchet: the artifact only ever moves uphill.
Two ways to use it
This repo is both a Claude Code skill and a standalone Python CLI. Same files, two entry points.
A) As a Claude Code skill (zero-config agent loop)
Clone the repo straight into your skills directory:
git clone https://github.com/Evarodenas/autoresearch ~/.claude/skills/autoresearch
That's it. The skill auto-activates when you say things like "optimize this",
"iterate on this", "find the best version", or "autoresearch this". The
agent reads SKILL.md, picks the right domain playbook from references/, and
drives the loop — committing keepers and reverting failures for you.
Works the same way in any agent runtime that loads
SKILL.md-style skills. Seereferences/tool-integration.mdfor the (small) set of capabilities the loop needs and how they map to other tools.
B) As a standalone CLI (no agent required)
The Python engine parses program.md, runs baseline evaluations, manages the
git ratchet, and generates a review dashboard:
# Establish a baseline score for your artifact
python scripts/autoresearch_runner.py --program program.md --baseline-only
# After running experiments, print a summary
python scripts/autoresearch_runner.py --program program.md --summary
# Generate a self-contained HTML review dashboard from results.tsv
python scripts/generate_review.py <workspace-dir> --static review.html
The CLI ships with built-in evaluators for content and code; for
prompt and model domains you supply a frozen evaluate.* (templates are
in the references/ playbooks).
Quickstart (60 seconds)
git clone https://github.com/Evarodenas/autoresearch
cd autoresearch
pip install -r requirements.txt # only needs `textstat` for the content demo
cd examples/quickstart
python ../../scripts/autoresearch_runner.py --program program.md --baseline-only
You'll see a baseline content score for draft.md. Now point your agent at this
folder and say "autoresearch draft.md using program.md" — it will iterate the
draft, keeping only edits that raise the score, and log every attempt to
results.tsv. Then render the dashboard:
python ../../scripts/generate_review.py . --static review.html && open review.html
Writing a program.md
Copy examples/program-template.md and fill it
in. A good one is short and specific:
## Goal
Maximize the content score of draft.md while keeping it under 1500 words.
## Target Artifact
`draft.md`
## Frozen Evaluation
Built-in content evaluator (composite readability + structure + voice score).
## Metric
Primary: composite content score (0–100).
## Constraints
- Stay under 1500 words
- Keep the three section headers
- Don't change the core argument
## Search Space
- Opening hook style
- Paragraph rhythm and length
- Subheading wording
- CTA placement
## Budget
8 experiments per session.
Domain playbooks
Each file in references/ has ready-to-use frozen-evaluation
scripts and a curated search space:
| Domain | Playbook | Good for |
|---|---|---|
| Code | code-loop.md | React/HTML/CSS, Python scripts, perf & a11y |
| Prompt | prompt-loop.md | system prompts, agent instructions, tool descriptions |
| Model/Config | model-loop.md | scoring weights, pipeline configs, backtested models |
| Content | content-loop.md | blog posts, social posts, copy, presentation scripts |
Plus review-dashboard.md (how to inspect
results) and tool-integration.md (how the
loop maps onto whatever tools you have).
The seven rules of the ratchet
- One change per experiment. Always.
- Log everything — failures included. They stop you re-trying dead ends.
- Prefer simpler solutions. Complexity is where bugs hide.
- Fixed budget per experiment. No single run hogs resources.
- Human edits
program.md, agent edits the target. Never cross it. - Revert cleanly. A failed experiment leaves the artifact exactly as before.
- Don't stop, don't ask. Run until the budget is spent or a human steps in.
Requirements
- Python 3.9+ and git for the CLI.
textstat(inrequirements.txt) for the built-in content evaluator.anthropiconly if you use the meta-judge in the prompt playbook.
Credit
The autoresearch loop — frozen prepare.py, mutable train.py, the human
programming program.md files instead of code — is Andrej Karpathy's idea.
This repo is an independent, runtime-agnostic implementation of that pattern.