Octopus: Orquestrador de Workflows para Agentes de IA em ML/DL/RAG
quanai06/octopus
Orquestrador baseline-first com guardrails para ML/RAG com agentes IA
Ver no GitHubCLI Python que estrutura projetos de ML/DL/RAG com baseline-first workflow, memória de experimentos, guardrails e integração MCP nativa para Claude Code/Codex. Mantém agentes de IA honestos através de gates de tarefa, contexto compacto e rastreamento de runs sistemático, evitando armadilhas comuns como ajuste em test set e perda de rastreabilidade.
Instalação
npx convez-ai quanai06/octopus1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Estruturar projetos de ML/DL com baseline obrigatório antes de tuning
- Manter contexto compacto e relevante para Claude Code/Codex
- Rastrear experimentos e garantir reprodutibilidade entre sessões
- Validar splits train/val/test e políticas de dados explicitamente
- Evitar mudanças não-rastreadas e tuning sem evidência
Veredito

Ferramenta madura e bem-documentada para equipes que trabalham com IA generativa em ciclos ML iterativos, especialmente com Codex/Claude. Reduz significativamente erros metodológicos e perda de contexto, mas exige disciplina do usuário e é overkill para notebooks rápidos. Arquitetura clara com extensibilidade via MCP.
README
A tradução para o português deste README ainda está na fila.
CLI Octopus
Why Use Octopus?
Serious ML/DL/RAG work fails less from lack of ideas than from messy process: agents skip the baseline, tune on the test set, change splits without noticing, forget which experiment was actually better, and burn context rereading the same planning files. Octopus exists to make those mistakes harder.
Octopus is a Python CLI for baseline-first AI-assisted ML projects. It gives Codex a small project brain: structured intake, fixed task gates, compact context, experiment memory, and baseline profiling. It does not replace your training code; it keeps the agent honest while it writes and runs that code.
Use Octopus when you care about:
- getting a real baseline before the main model,
- keeping train/validation/test policy explicit,
- giving Codex only the context it needs instead of the whole repo,
- logging runs so future tuning is based on evidence,
- resuming work without rediscovering the project every session,
- making one controlled improvement at a time.
Do not use Octopus if you only need a quick notebook experiment and do not care about reproducibility, experiment memory, or agent guardrails.
The normal loop is:
intake -> baseline spec/context -> baseline run -> ingest/profile
-> select next direction -> one controlled improvement
For fast baselines, use baseline_spec.yaml and minimal-baseline context. For
longer projects, use the full planning/task/experiment loop.
📚 Full documentation: docs/ — getting started, concepts,
how-to guides (Claude Code, Codex, headless, tracker ingest, tuning loop, resume),
and a complete CLI / configuration / files reference.
Project docs:
- Architecture — system map, lifecycle, state files, tools, MCP, and extension points.
- Contributing — development setup, testing, PR checklist, and contribution guidelines.
What Octopus Gives Codex
- A Codex skill:
@octopus-baseline. - A small fast-path baseline contract:
baseline_spec.yaml. - A compact working file:
.octopus/context/current_context.md. - A minimal context profile for quick baseline work.
- Baseline-first task gates in
.octopus/tasks.json. - Experiment memory and profiling under
.octopus/experiments/. - Prompt-router fallbacks under
~/.codex/prompts/.
Official Codex references:
- Codex docs: https://developers.openai.com/codex
- Codex CLI docs: https://developers.openai.com/codex/cli
- Codex use cases: https://developers.openai.com/codex/use-cases
Install
Install the released package from PyPI:
python -m pip install cli-octopus
octopus --help
Python 3.11+ is required.
Install the latest GitHub version without waiting for a PyPI release:
python -m pip install git+https://github.com/quanai06/octopus.git
For local development:
git clone https://github.com/quanai06/octopus.git
cd octopus
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
Verify the dev install:
octopus --help
pytest
Install Codex CLI if you do not already have it:
curl -fsSL https://chatgpt.com/codex/install.sh | sh
codex
Fast Start With Codex
Install once:
python -m pip install -U cli-octopus
octopus install --runtime codex
Use in any ML/DL/RAG project:
cd your-ml-project
codex
In Codex, choose the skill:
/skills
Select Octopus Baseline. Or invoke it directly:
@octopus-baseline run the baseline workflow for this project
Restart Codex after octopus install --runtime codex if Codex was already open.
The skill initializes Octopus when needed, builds .octopus/context/current_context.md,
and prepares the first baseline plan/script before any main-model work.
If your Codex surface does not load skills, open or paste the fallback prompt:
cat ~/.codex/prompts/octopus-baseline.md
Uninstall:
octopus uninstall --runtime codex
Manual Codex Setup
Use this when you want full control instead of the one-shot octopus-baseline
router.
mkdir viet-emotion-classifier
cd viet-emotion-classifier
octopus init --runtime codex
octopus ask
octopus plan
octopus ml-plan
octopus tasks
octopus task next
octopus context --task "train the baseline" --profile training
codex
Prompt Codex:
This project uses Octopus. Run `octopus task next`, then read ONLY
`.octopus/context/current_context.md` as your working context.
Implement the baseline first and stop after writing the baseline training-script
skeleton. Do not train unless I explicitly ask. Do not start the main model
before a baseline, do not change the train/validation/test split, and do not tune
on the test set.
Headless Setup For Codex
octopus ask is interactive. For Codex, CI, or benchmark runs, use a YAML/JSON
answers file.
Example answers.yaml:
project_name: VSMEC Emotion Classifier
project_goal: Build a Vietnamese social-media emotion classifier.
target_users: ML engineers
project_type: machine learning
task_type: text_classification
input_type: text
output_type: emotion_label
dataset_status: available
dataset_size_note: tests/datasets/vsmec, fixed train/valid/test XLSX files
has_labels: true
has_class_imbalance: true
main_metric: macro_f1
baseline_model: TF-IDF + Logistic Regression
runtime:
- codex
compute:
has_gpu: false
budget_note: CPU only
Run:
octopus init --runtime codex --force
octopus ask --from answers.yaml
octopus plan
octopus ml-plan
octopus tasks
octopus context --task "train the baseline" --profile training
Then ask Codex to read only:
.octopus/context/current_context.md
Use With Claude Code
Octopus installs the same baseline-first workflow into Claude Code, plus two
things the Codex surface does not get: specialized subagents and a hard
PreToolUse guard.
Install the Claude Code artifacts:
octopus install --runtime claude
This writes:
~/.claude/commands/octopus-baseline.md
~/.claude/commands/octopus-plan.md
~/.claude/commands/octopus-train.md
~/.claude/commands/octopus-tune.md
~/.claude/commands/octopus-status.md
~/.claude/commands/octopus-resume.md
~/.claude/agents/octopus-baseline-runner.md
~/.claude/agents/octopus-experiment-analyst.md
~/.claude/agents/octopus-tuner.md
~/.claude/agents/octopus-data-auditor.md
~/.claude/agents/octopus-rag-evaluator.md
~/.claude/settings.json # baseline-guard PreToolUse hook (merged, idempotent)
~/.claude/.octopus-manifest.json
Existing settings.json keys are preserved, and reinstalling never duplicates
the hook.
Fast start
From inside your ML/DL/RAG project:
octopus install --runtime claude
claude
In Claude Code, run the slash command:
/octopus-baseline
It follows the same path as Codex: initialize if needed, collect missing facts
and run octopus ask --from answers.yaml, then octopus plan --force,
octopus ml-plan --force, octopus tasks --force, octopus task next,
octopus context --task "train the baseline" --profile training, read
.octopus/context/current_context.md, write the baseline plan + script skeleton,
and stop before training.
Other slash commands: /octopus-plan, /octopus-train, /octopus-tune,
/octopus-status, /octopus-resume.
Subagents
Claude Code can delegate to the installed subagents:
octopus-baseline-runner— establish and ingest the first baseline.octopus-experiment-analyst— analyze/profile a finished run.octopus-tuner— implement exactly one selected direction.octopus-data-auditor— split / leakage / imbalance audit.octopus-rag-evaluator— retrieval eval + citation/faithfulness.
Baseline-guard hook
A PreToolUse hook on the Bash tool blocks main-model training before a
completed baseline exists. Commands like python train.py,
accelerate launch train.py, torchrun ..., python train_phobert.py, or any
fine-tune command exit with code 2 and an explanation until you log a
baseline. The baseline scripts themselves are not blocked.
Manual setup
Use this for full control instead of the one-shot /octopus-baseline:
octopus init --runtime claude
octopus ask # or: octopus ask --from answers.yaml (headless)
octopus plan
octopus ml-plan
octopus tasks
octopus task next
octopus context --task "train the baseline" --profile training
claude
Then prompt Claude Code:
This project uses Octopus. Run `octopus task next`, then read ONLY
`.octopus/context/current_context.md` as your working context.
Implement the baseline first and stop after writing the baseline training-script
skeleton. Do not train unless I ask. Do not start the main model before a
baseline, do not change the train/validation/test split, and do not tune on the
test set.
Uninstall:
octopus uninstall --runtime claude
Install both runtimes at once with octopus install --runtime claude,codex.
Commands
Core project setup:
octopus init # create .octopus/ plus generated files
octopus ask # interactive intake, or --from answers.yaml
octopus plan # render requirements.md
octopus ml-plan # render ml_design/data/compute/experiment plans
octopus tasks # render tasks.md and .octopus/tasks.json
octopus sync # refresh AGENTS.md / CLAUDE.md from current state
octopus status # show project snapshot
Task/context commands:
octopus task next
octopus task start T010
octopus task done T010
octopus context --task "train the baseline" --profile training
octopus context --direction D1 --target codex
Experiment commands:
octopus exp log --kind baseline --name baseline --metric macro_f1=0.58
octopus exp ingest --run-dir <run_dir> --kind baseline
octopus exp profile
octopus exp next
octopus exp choose D1
octopus exp compare --metric macro_f1
octopus exp report
Structured tools and MCP:
octopus status --json
octopus task next --json
octopus context --task "train the baseline" --profile training --json
octopus exp ingest --run-dir <run_dir> --kind baseline --json
octopus exp profile --json
octopus tool list --json
octopus tool call octopus_status
octopus tool call octopus_build_context --input-json '{"task":"train baseline"}'
octopus mcp # MCP stdio server
The structured tools expose JSON schemas for function-calling style agents:
octopus_status, octopus_task_next, octopus_build_context,
octopus_ingest_run, and octopus_profile_baseline. The MCP server exposes the
same tools plus resources such as octopus://context/current,
octopus://memory/experiments, octopus://session/current, and
octopus://reports/baseline_profile.
Runtime install:
octopus install --runtime codex
octopus uninstall --runtime codex
Generated Files
Project files:
requirements.md
ml_design.md
experiment_plan.md
data_strategy.md
compute_budget.md
tasks.md
AGENTS.md
Octopus state:
.octopus/config.yaml
.octopus/project_state.json
.octopus/tasks.json
.octopus/context/current_context.md
.octopus/experiments/
.octopus/reports/
.octopus/plans/
.octopus/session/
.octopus/memory/
Codex prompt routers:
~/.codex/prompts/octopus-baseline.md
~/.codex/prompts/octopus-plan.md
~/.codex/prompts/octopus-train.md
~/.codex/prompts/octopus-tune.md
~/.codex/prompts/octopus-status.md
~/.codex/prompts/octopus-resume.md
Baseline-First Rules
For ML/DL/RAG projects, Octopus enforces these rules:
- Start with a simple reproducible baseline.
- Do not log or start main-model work before a completed baseline exists.
- Do not tune on the test set.
- Do not change train/validation/test split unless the selected direction says so.
- Change one thing per experiment.
- Track the project's main metric and diagnostic metrics, not just accuracy.
Logging or ingesting a completed baseline marks T010, T011, and T012 done,
then unblocks T020.
What "baseline" means here
The baseline is a full, data-type-aware train/eval protocol — not a single
random split. octopus ml-plan renders a "Split & Cross-Validation" section in
data_strategy.md and experiment_plan.md, chosen by task type:
| Task | Split + CV |
|---|---|
| text / image classification | canonical cleaned dataset manifest; StratifiedKFold(k=5) + per-class recall, mean ± std; StratifiedGroupKFold if rows share a group; for deep models use k-fold when feasible or ≥3 fixed seeds with an explicit exception |
| regression (tabular) | KFold(k=5); GroupKFold if rows share an entity; TimeSeriesSplit if the target is time-ordered |
| forecasting | TimeSeriesSplit (expanding/rolling), never shuffle, lag features computed inside each fold, compared to a naive baseline |
| retrieval / RAG | canonical cleaned corpus manifest; fixed labeled query eval set; documented chunking grid; fixed top-k grid; Recall@k / MRR / source-hit; rerank only later over a recorded candidate pool; never tune on test queries |
| recommendation | time-aware split on future interactions; guard cold-start and leakage |
| other | default: held-out test + k-fold (fold scheme chosen to match the data) |
Across all of them: preprocessing is fit inside each fold (leakage-safe), the held-out test set stays untouched until final reporting, and the main metric is reported as mean ± std across folds. The model can be simple; the protocol is rigorous.
Example:
octopus exp ingest --run-dir runs/E001 --kind baseline
octopus exp profile
octopus exp next
octopus exp choose D1
octopus context --direction D1 --target codex
Then in Codex:
Read `.octopus/context/current_context.md` and implement only selected direction D1.
Do not implement multiple directions at once. Stop before training unless I ask.
Baseline Profile And Tuning
After a baseline is logged:
octopus exp profile
Octopus writes:
.octopus/reports/baseline_profile.md
The profile reports:
- baseline standing vs target;
- weak classes or low retrieval metrics;
- bias/variance symptoms;
- data quality flags;
- recommended techniques;
- "Do Not Try Yet" items.
Then:
octopus exp next
octopus exp choose D1
octopus context --direction D1 --target codex
This gives Codex a smaller direction-specific context instead of the whole planning history.
Tracker Auto-Ingest
octopus exp ingest can read common tracker output directories:
octopus exp ingest --run-dir mlruns/0/<run_id> --kind baseline # MLflow
octopus exp ingest --run-dir wandb/run-<id> # W&B
octopus exp ingest --run-dir runs/<name> # TensorBoard
octopus exp ingest --run-dir runs/E001 --tracker none # plain files
Plain run directories can include:
metrics.json
classification_report.json
config.yaml
Session Resume
Use this when Codex loses context or starts a new session:
octopus session start --goal "beat the baseline"
octopus session show
octopus resume
Then in Codex:
octopus-resume
or paste:
cat ~/.codex/prompts/octopus-resume.md
Benchmarks
Benchmarks are deterministic local token measurements using Octopus'
cl100k_base estimator. They do not train models.
Baseline Plan + Script Skeleton
python tests/benchmark/token_eval_datasets.py
Datasets:
| Scenario | Dataset | Task |
|---|---|---|
| ML | tests/datasets/vsmec | Vietnamese emotion classification |
| DL | tests/datasets/alpaca-dataset/dataset | Alpaca / not-alpaca image classification |
| RAG | tests/datasets/wikiqa | BM25 retrieval evaluation |
Latest local result:
| Scenario | A prompt-only input | B Octopus input | Saving | Output plan+script |
|---|---|---|---|---|
| ML | 3,733 | 2,750 | 26.3% | 708 |
| DL | 2,437 | 2,027 | 16.8% | 397 |
| RAG | 2,512 | 2,027 | 19.3% | 432 |
Post-Baseline Stacking / Fusion Upgrade
python tests/benchmark/token_eval_post_baseline.py
Latest local result:
| Scenario | A prompt-only input | B Octopus direction input | Saving | Output plan+script |
|---|---|---|---|---|
| ML | 5,001 | 1,222 | 75.6% | 270 |
| DL | 3,531 | 1,214 | 65.6% | 274 |
| RAG | 3,611 | 1,196 | 66.9% | 271 |
Interpretation:
- First baseline turn saves moderately because Codex still needs project facts.
- Post-baseline tuning saves much more because Octopus compresses state into a selected direction, evidence, guardrails, and relevant code context.
Agent-dependent live token usage is separate. Measure it from Codex' own usage counter in fresh sessions and score the produced artifacts with the rubric in:
eval_token_and_compliance.md
Troubleshooting
octopus command not found:
source .venv/bin/activate
python -m pip install cli-octopus
octopus --help
For a local checkout:
python -m pip install -e ".[dev]"
python -m octopus.cli.main --help
Codex does not recognize octopus-baseline:
octopus install --runtime codex
ls ~/.codex/prompts
cat ~/.codex/prompts/octopus-baseline.md
Then paste the prompt content into Codex.
Codex tries to skip the baseline:
Read `.octopus/context/current_context.md`. Follow Octopus baseline-first rules.
Do not start the main model before a completed baseline exists.
No baseline exists but you want to tune:
octopus task next
octopus context --task "train the baseline" --profile training
After the real baseline run:
octopus exp ingest --run-dir <run_dir> --kind baseline
octopus exp profile
CI/CD
GitHub Actions workflows live in .github/workflows/.
CI runs on push to main, pull requests to main, and manual dispatch:
.github/workflows/ci.yml
CI jobs:
- Python 3.11 and 3.12 run the same local gate as developers:
make check
- A package build job verifies source distribution and wheel creation:
python -m build
Publishing is release-driven:
.github/workflows/publish.yml
It builds the package and publishes to PyPI when a GitHub Release is published.
The workflow uses PyPI Trusted Publishing (id-token: write), so configure the
PyPI project to trust:
owner: quanai06
repository: octopus
workflow: publish.yml
environment: pypi
Release flow:
make check
python -m build
git tag v0.1.1
git push origin v0.1.1
Then create/publish the GitHub Release for that tag. The publish workflow will upload the package to PyPI.
PyPI never allows re-uploading the same version. If the workflow fails with
File already exists, that version is already published. Bump pyproject.toml
to the next unused version:
version = "0.1.2"
then commit, tag, and publish a new GitHub Release.
After publish, test the package from a clean environment:
python -m venv /tmp/octopus-test
source /tmp/octopus-test/bin/activate
python -m pip install cli-octopus
octopus --help
octopus tool list --json
License
MIT. See LICENSE.
Development
python tests/benchmark/token_eval_datasets.py
python tests/benchmark/token_eval_post_baseline.py
pytest
ruff check .
mypy src
This README is Codex-first, but Claude Code is fully supported — see Use With Claude Code for slash commands, subagents, and the baseline-guard hook.