convez.ai
SkillAutomaçãoIntermediário
Evarodenas

AutoResearch

Evarodenas/autoresearch

Loops de otimização autônoma com ratchet e git para code, prompts e configs

Ver no GitHub
InicialPluginScore · de 0 a 100

Skill para otimização autônoma e iterativa de qualquer artefato (código, prompts, conteúdo, modelos) usando o padrão de Karpathy. Define métricas congeladas em program.md e deixa um agente IA executar experimentos com git como histórico — cada commit mantido é uma melhoria genuína.

Instalação

$npx convez-ai Evarodenas/autoresearch

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
57%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Otimizar prompts de IA iterativamente com feedback automático
  • Melhorar código mediante experimentos quantificados e versionados
  • Iterar em conteúdo ou configurações com métrica congelada
  • Pesquisa de hiperparâmetros ou ajuste de modelos sem intervenção manual

Veredito

Implementação sólida e bem-arquitetada do padrão autoresearch de Karpathy. Segue boas práticas (métrica congelada, ratchet via git, histórico limpo) e oferece flexibilidade como skill Claude e CLI standalone. Útil para quem quer pesquisa/otimização sistemática e auditável.

README

A tradução para o português deste README ainda está na fila.

autoresearch

Autonomous, ratcheting optimization loops for any artifact — an open implementation of Andrej Karpathy's "autoresearch" pattern.

You write a program.md describing what "better" means. An AI agent then runs experiments on your behalf: form a hypothesis → make one change → score it against a frozen metric → keep it if it improved (git commit) or throw it away if it didn't (git reset --hard). Every surviving commit is a genuine improvement. The git history is the research record.

Works on code, prompts, content, models, and configs — anything you can attach a number to.

┌──────────────┐   writes    ┌──────────────┐   reads &    ┌──────────────┐
│    HUMAN     │ ─────────▶  │  program.md  │  executes ▶  │  AI  AGENT   │
└──────────────┘             └──────────────┘              └──────┬───────┘
                                                                  │ one change
   keep ◀── better? ── run frozen eval ◀── git commit ◀──────────┘
   revert ◀ worse/crash ── git reset --hard

Why this design works

FileRoleWho edits it
program.mdThe research spec: goal, metric, constraints, search space, budgetYou
target artifactThe thing being optimizedThe agent
evaluate.*How success is scoredFrozen — nobody touches it mid-run
results.tsvAppend-only experiment logThe agent appends

The wall between the frozen evaluation and the mutable target is the whole trick. The agent can't cheat by editing how it's graded, so it's safe to let it run unattended. Git enforces a monotonic ratchet: the artifact only ever moves uphill.


Two ways to use it

This repo is both a Claude Code skill and a standalone Python CLI. Same files, two entry points.

A) As a Claude Code skill (zero-config agent loop)

Clone the repo straight into your skills directory:

git clone https://github.com/Evarodenas/autoresearch ~/.claude/skills/autoresearch

That's it. The skill auto-activates when you say things like "optimize this", "iterate on this", "find the best version", or "autoresearch this". The agent reads SKILL.md, picks the right domain playbook from references/, and drives the loop — committing keepers and reverting failures for you.

Works the same way in any agent runtime that loads SKILL.md-style skills. See references/tool-integration.md for the (small) set of capabilities the loop needs and how they map to other tools.

B) As a standalone CLI (no agent required)

The Python engine parses program.md, runs baseline evaluations, manages the git ratchet, and generates a review dashboard:

# Establish a baseline score for your artifact
python scripts/autoresearch_runner.py --program program.md --baseline-only

# After running experiments, print a summary
python scripts/autoresearch_runner.py --program program.md --summary

# Generate a self-contained HTML review dashboard from results.tsv
python scripts/generate_review.py <workspace-dir> --static review.html

The CLI ships with built-in evaluators for content and code; for prompt and model domains you supply a frozen evaluate.* (templates are in the references/ playbooks).


Quickstart (60 seconds)

git clone https://github.com/Evarodenas/autoresearch
cd autoresearch
pip install -r requirements.txt      # only needs `textstat` for the content demo

cd examples/quickstart
python ../../scripts/autoresearch_runner.py --program program.md --baseline-only

You'll see a baseline content score for draft.md. Now point your agent at this folder and say "autoresearch draft.md using program.md" — it will iterate the draft, keeping only edits that raise the score, and log every attempt to results.tsv. Then render the dashboard:

python ../../scripts/generate_review.py . --static review.html && open review.html

Writing a program.md

Copy examples/program-template.md and fill it in. A good one is short and specific:

## Goal
Maximize the content score of draft.md while keeping it under 1500 words.

## Target Artifact
`draft.md`

## Frozen Evaluation
Built-in content evaluator (composite readability + structure + voice score).

## Metric
Primary: composite content score (0–100).

## Constraints
- Stay under 1500 words
- Keep the three section headers
- Don't change the core argument

## Search Space
- Opening hook style
- Paragraph rhythm and length
- Subheading wording
- CTA placement

## Budget
8 experiments per session.

Domain playbooks

Each file in references/ has ready-to-use frozen-evaluation scripts and a curated search space:

DomainPlaybookGood for
Codecode-loop.mdReact/HTML/CSS, Python scripts, perf & a11y
Promptprompt-loop.mdsystem prompts, agent instructions, tool descriptions
Model/Configmodel-loop.mdscoring weights, pipeline configs, backtested models
Contentcontent-loop.mdblog posts, social posts, copy, presentation scripts

Plus review-dashboard.md (how to inspect results) and tool-integration.md (how the loop maps onto whatever tools you have).


The seven rules of the ratchet

  1. One change per experiment. Always.
  2. Log everything — failures included. They stop you re-trying dead ends.
  3. Prefer simpler solutions. Complexity is where bugs hide.
  4. Fixed budget per experiment. No single run hogs resources.
  5. Human edits program.md, agent edits the target. Never cross it.
  6. Revert cleanly. A failed experiment leaves the artifact exactly as before.
  7. Don't stop, don't ask. Run until the budget is spent or a human steps in.

Requirements

  • Python 3.9+ and git for the CLI.
  • textstat (in requirements.txt) for the built-in content evaluator.
  • anthropic only if you use the meta-judge in the prompt playbook.

Credit

The autoresearch loop — frozen prepare.py, mutable train.py, the human programming program.md files instead of code — is Andrej Karpathy's idea. This repo is an independent, runtime-agnostic implementation of that pattern.

License

MIT

Ver o original no GitHub

Bom para

Relacionados