convez.ai
AgenteAutomaçãoIntermediário
jonsams968-cloud

OnHands3 — Cursor Inteligente com IA

jonsams968-cloud/onhands3

Automatize desktop com voz e texto via Claude Code Agent

Ver no GitHub
InicialPluginScore · de 0 a 100

Agent de desktop baseado em Claude Code que interpreta comandos de voz e texto para automatizar tarefas no Windows 11. Integra reconhecimento de fala em tempo real (Tencent Cloud ou Whisper local), roteamento inteligente entre Direct AI e Agent CLI, e contexto automático (arquivos selecionados, janela ativa, clipboard). Executa operações de arquivo, geração de mídia e tarefas de codificação com interface de streaming.

Instalação

$npx convez-ai jonsams968-cloud/onhands3

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
2
Forks
0
Acompanhando
0
Saúde
28%
Último commit
há 4 meses
Versão
v0.7.7
Issues abertas
0
Contribuidores
2

Casos de uso

  • Automatizar operações de arquivos (mover, renomear, organizar) com comandos de voz
  • Gerar imagens/vídeos e fazer processamento de mídia sob demanda
  • Executar tarefas de codificação e scripts via Agent CLI com contexto automático
  • Tradução e processamento de texto selecionado sem deixar o contexto atual
  • Integração de IA em fluxos de trabalho desktop existentes via overlay inteligente

Veredito

Projeto técnico sólido que traz automação de desktop para Windows via Claude Code Agent. Bom ponto de partida para quem busca integrar agents IA em workflows de produtividade local, com suporte a contexto rico e roteamento inteligente, mas limitado ao Windows 11 e dependente de configuração de chaves API externas.

README

A tradução para o português deste README ainda está na fila.


What It Does

InputExampleExecution
🎤 Long press + voice"Move these files to a new folder"Agent (Claude Code)
⌨️ Text input"Translate to Chinese"Agent / Direct AI
📂 Select files + long press"Rename with today's date"Agent (auto-detects selection)

Key capabilities:

  • Voice recognition — Tencent Cloud real-time ASR (streaming), local Whisper fallback, good Chinese accuracy
  • Smart routing — simple Q&A via Direct AI, file ops/coding via Agent CLI
  • Context awareness — foreground window, working directory, selected files, selected text, clipboard
  • Real-time streaming — tool calls and text output appear as the agent works
  • ASK protocol — agent can present choices to the user when intent is ambiguous
  • Media generation — image/video generation with preview UI
  • Global interrupt — hold ESC 5s to force kill anytime

What It Does / 功能概览

输入方式示例指令执行方式
🎤 长按鼠标 + 语音"把这些文件移到新建文件夹"Agent (Claude Code)
⌨️ 文本输入"翻译成中文"Agent / Direct AI
📂 选中文件 + 长按"重命名为今天的日期"Agent(自动识别选中文件)
  • 语音识别 — 腾讯云实时 ASR(流式),本地 Whisper fallback
  • 智能路由 — 简单问答走 Direct AI,文件操作/编程走 Agent CLI
  • 上下文感知 — 自动收集前台窗口、工作目录、选中文件、选中文字、剪贴板
  • 实时流 — Agent 执行过程实时显示
  • ASK 协议 — Agent 无法判断意图时弹出选项让用户选择
  • 媒体生成 — 支持图片/视频生成,带预览 UI
  • 全局中断 — 长按 ESC 5 秒强制退出

System Requirements / 系统要求

ItemRequirement
OSWindows 11 (x64)
Node.js>= 18.x
Agent CLIClaude Code (recommended) or Codex / OpenCode
GPUOptional — local Whisper can use CPU (slower) or CUDA
MicrophoneRequired — for voice input

Windows only for now. macOS/Linux needs MouseMonitor Win32 API adaptation.


Installation / 安装

1. Clone / 克隆

git clone https://github.com/jonsams968-cloud/onhands3.git
cd onhands3
npm install

2. Install Agent CLI / 安装 Agent CLI

npm install -g @anthropic-ai/claude-code
claude --version

3. Configure / 配置

cp .env.example .env

Edit .env:

# AI API (Direct AI fast mode)
AI_API_KEY=your-api-key
AI_BASE_URL=https://apihub.agnes-ai.com/v1
AI_MODEL=agnes-2.0-flash

# Speech-to-text
STT_MODE=tencent
WHISPER_MODEL=large-v3-turbo

# Tencent Cloud ASR (for STT_MODE=tencent)
TENCENT_SECRET_ID=
TENCENT_SECRET_KEY=
TENCENT_APP_ID=

# Long press sensitivity (ms)
LONG_PRESS_DURATION=800

# Force all requests through Agent
FORCE_AGENT=true

4. Download Whisper Model / 下载 Whisper 模型

For STT_MODE=local, download the model file:

Manual placement:

%APPDATA%/onhands3/data/whisper/
├── Release/
│   └── whisper-cli.exe
└── ggml-large-v3-turbo.bin

Usage / 使用

Dev Mode / 开发模式

npm run dev

If launching from inside an Electron app (e.g. CherryStudio), run unset ELECTRON_RUN_AS_NODE first.

Controls / 操作

  1. Long press (default 800ms) → start recording → release to transcribe
  2. Click input button → switch to text input mode
  3. Hold ESC 5s → force kill
  4. Test shortcuts:
    • Ctrl+Shift+1 — simulate recording
    • Ctrl+Shift+2 — simulate transcription
    • Ctrl+Shift+3 — simulate routing
    • Ctrl+Shift+4 — simulate execution with streaming
    • Ctrl+Shift+0 — hide window

Build / 构建

npm run build
npm run preview

Architecture / 项目架构

src/
├── main/                          # Electron main process
│   ├── index.ts                   # Window creation, global shortcuts, IPC
│   ├── config.ts                  # Environment config loader
│   ├── orchestrator/
│   │   └── Orchestrator.ts        # Core: voice → route → execute → result
│   ├── agents/
│   │   ├── ClaudeCodeAgent.ts     # Claude Code CLI wrapper (stream-json)
│   │   └── AgentDetector.ts       # Auto-detect installed Agent CLIs
│   ├── ai/
│   │   ├── DirectAI.ts            # Direct AI API (quick tasks)
│   │   └── Router.ts              # Command routing: direct vs agent
│   ├── input/
│   │   ├── MouseMonitor.ts        # Mouse long-press detection (GetAsyncKeyState poll)
│   │   └── SelectionMonitor.ts    # Text selection capture (child process + selection-hook)
│   ├── context/
│   │   └── ContextCollector.ts    # Context: window, selected files/text, clipboard
│   ├── stt/
│   │   ├── WhisperSTT.ts          # STT factory: local / cloud / tencent
│   │   └── TencentASR.ts          # Tencent Cloud real-time ASR (WebSocket)
│   └── permission/
│       └── PermissionServer.ts    # Dangerous operation approval system
├── renderer/                      # Renderer process (React)
│   ├── App.tsx                    # UI state machine + glassmorphism overlay
│   ├── hooks/useVoiceRecorder.ts  # Microphone recording (MediaRecorder API)
│   └── styles.css                 # All styles (no Tailwind)
├── preload/
│   └── index.ts                   # IPC bridge (contextBridge)
├── shared/
│   └── types.ts                   # Shared type definitions
scripts/
└── selection-worker.cjs           # selection-hook worker (runs in child process)

Execution Flow / 执行流程

Long press → hide overlay → capture foreground window → show recording UI
    ↓
Release → audio data via IPC → Whisper STT transcription
    ↓
Route (Direct AI / Agent CLI)
    ↓
Collect context (window, selected text, files, clipboard, working dir)
    ↓
Execute → stream output → show result / auto-hide after 12s

Tech Stack / 技术栈

LayerTechnology
FrameworkElectron 35 + electron-vite 3
UIReact 18 + TypeScript 5.7
STTTencent Cloud ASR (streaming) / whisper.cpp (local fallback) / OpenAI Whisper API (cloud)
AgentClaude Code CLI (stream-json mode)
Text selectionselection-hook via child process (UIA + IAccessible + Clipboard fallback)
Window captureWin32 API via koffi (GetForegroundWindow, Shell COM)
Mouse inputWin32 API via koffi (GetAsyncKeyState polling)
Audioffmpeg-static (webm → wav conversion)

Config Reference / 配置说明

VariableDescriptionDefault
AI_API_KEYDirect AI API key-
AI_BASE_URLAI API endpointhttps://apihub.agnes-ai.com/v1
AI_MODELAI model nameagnes-2.0-flash
STT_MODESTT mode: local / cloud / tencentcloud
WHISPER_MODELWhisper model sizelarge-v3-turbo
LONG_PRESS_DURATIONLong press threshold (ms)800
DRAG_THRESHOLD_PXDrag detection threshold (px)15
FORCE_AGENTForce all requests to Agentfalse

Known Issues / 已知问题

  • White title bar flash — Electron 35 bug (#47946), DWM briefly draws white chrome on focus change. Fixed in Electron 37.3.1+
  • Agent encoding — Chinese path operations occasionally hit UTF-8 issues. Mitigated with single-quote wrapping + UTF-8 prefix
  • Windows only — MouseMonitor uses Win32 API. macOS/Linux needs reimplementation
  • GPU cache — If app exits immediately with GPU cache errors, delete %LOCALAPPDATA%/OnHands3/GPUCache and restart

Version History / 版本历史

VersionDateSummary
v0.482026-06-09Tencent Cloud streaming ASR, Phase 1 stability (history, parallel capture, crash recovery), permission dialog fix
v0.472026-06-08selection-hook replaces Ctrl+C for text selection capture
v0.462026-06-07ASK protocol — agent can present choices to user
v0.452026-06-07Permission API with proper Chinese encoding
v0.442026-06-06Permission server + Shell COM merge
v0.432026-06-06Fix foreground capture on re-trigger + img2img mode
v0.422026-06-05Universal media preview + selected image context

Roadmap / 路线图

Phase 1 — Core Experience / 基础体验 ✅

Make it fast enough and stable enough that people want to use it daily.

#ItemDescriptionStatus
1.1Speed / 速度优化Parallel context collection, Tencent Cloud streaming STT, 5-min processing timeout✅ Done
1.2Stability / 稳定性Agent crash recovery, STT auto-fallback (tencent→local), permission dialog fix, network error handling✅ Done
1.3Recent History / 近期历史In-memory ring buffer (last 5 commands + result summaries), injected into prompt✅ Done

Phase 2 — Context Intelligence / 上下文智能

Richer context → fewer ASK prompts → higher success rate per invocation.

#ItemDescription
2.1Timestamp / 时间上下文Current time, day of week, date. Handles "today", "tomorrow", "next Monday"
2.2UIA Element Detection / 元素检测Detect element type at long-press position (image/link/button/input). Get src/href/alt text
2.3Screen OCR / 屏幕文字识别OCR screenshot text into context directly. Saves one agent tool-call round
2.4User Preferences / 用户偏好preferences.json: language, saveDir, commonProjects. Injected into prompt for consistent behavior

Phase 3 — UI & Usability / 界面与可用性

Polish after the foundation is solid.

#ItemDescription
3.1Markdown Rendering / Markdown 渲染Headings, lists, code blocks (syntax highlight), tables, bold/italic
3.2Result Layout / 结果排版Spacing, font sizing, long-text fold/scroll
3.3Settings Page / 前端配置页GUI for API Key, STT mode, long-press duration. Replace manual .env editing

Phase 4 — Ecosystem / 开放生态

Requires a user base to justify.

#ItemDescription
4.1SDK & Docs / SDK 文档Write after API freeze: IPC protocol, Agent protocol, ASK protocol
4.2Module Marketplace / 模块市场Plugin API + frontend marketplace page

Phases are soft boundaries — items can move across phases based on actual need.


Acknowledgments / 致谢


License

MIT

Ver o original no GitHub

Bom para

Relacionados