OnHands3 — Cursor Inteligente com IA
jonsams968-cloud/onhands3
Automatize desktop com voz e texto via Claude Code Agent
Ver no GitHubAgent de desktop baseado em Claude Code que interpreta comandos de voz e texto para automatizar tarefas no Windows 11. Integra reconhecimento de fala em tempo real (Tencent Cloud ou Whisper local), roteamento inteligente entre Direct AI e Agent CLI, e contexto automático (arquivos selecionados, janela ativa, clipboard). Executa operações de arquivo, geração de mídia e tarefas de codificação com interface de streaming.
Instalação
npx convez-ai jonsams968-cloud/onhands31 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Automatizar operações de arquivos (mover, renomear, organizar) com comandos de voz
- Gerar imagens/vídeos e fazer processamento de mídia sob demanda
- Executar tarefas de codificação e scripts via Agent CLI com contexto automático
- Tradução e processamento de texto selecionado sem deixar o contexto atual
- Integração de IA em fluxos de trabalho desktop existentes via overlay inteligente
Veredito

Projeto técnico sólido que traz automação de desktop para Windows via Claude Code Agent. Bom ponto de partida para quem busca integrar agents IA em workflows de produtividade local, com suporte a contexto rico e roteamento inteligente, mas limitado ao Windows 11 e dependente de configuração de chaves API externas.
README
A tradução para o português deste README ainda está na fila.
What It Does
| Input | Example | Execution |
|---|---|---|
| 🎤 Long press + voice | "Move these files to a new folder" | Agent (Claude Code) |
| ⌨️ Text input | "Translate to Chinese" | Agent / Direct AI |
| 📂 Select files + long press | "Rename with today's date" | Agent (auto-detects selection) |
Key capabilities:
- Voice recognition — Tencent Cloud real-time ASR (streaming), local Whisper fallback, good Chinese accuracy
- Smart routing — simple Q&A via Direct AI, file ops/coding via Agent CLI
- Context awareness — foreground window, working directory, selected files, selected text, clipboard
- Real-time streaming — tool calls and text output appear as the agent works
- ASK protocol — agent can present choices to the user when intent is ambiguous
- Media generation — image/video generation with preview UI
- Global interrupt — hold ESC 5s to force kill anytime
What It Does / 功能概览
| 输入方式 | 示例指令 | 执行方式 |
|---|---|---|
| 🎤 长按鼠标 + 语音 | "把这些文件移到新建文件夹" | Agent (Claude Code) |
| ⌨️ 文本输入 | "翻译成中文" | Agent / Direct AI |
| 📂 选中文件 + 长按 | "重命名为今天的日期" | Agent(自动识别选中文件) |
- 语音识别 — 腾讯云实时 ASR(流式),本地 Whisper fallback
- 智能路由 — 简单问答走 Direct AI,文件操作/编程走 Agent CLI
- 上下文感知 — 自动收集前台窗口、工作目录、选中文件、选中文字、剪贴板
- 实时流 — Agent 执行过程实时显示
- ASK 协议 — Agent 无法判断意图时弹出选项让用户选择
- 媒体生成 — 支持图片/视频生成,带预览 UI
- 全局中断 — 长按 ESC 5 秒强制退出
System Requirements / 系统要求
| Item | Requirement |
|---|---|
| OS | Windows 11 (x64) |
| Node.js | >= 18.x |
| Agent CLI | Claude Code (recommended) or Codex / OpenCode |
| GPU | Optional — local Whisper can use CPU (slower) or CUDA |
| Microphone | Required — for voice input |
Windows only for now. macOS/Linux needs MouseMonitor Win32 API adaptation.
Installation / 安装
1. Clone / 克隆
git clone https://github.com/jonsams968-cloud/onhands3.git
cd onhands3
npm install
2. Install Agent CLI / 安装 Agent CLI
npm install -g @anthropic-ai/claude-code
claude --version
3. Configure / 配置
cp .env.example .env
Edit .env:
# AI API (Direct AI fast mode)
AI_API_KEY=your-api-key
AI_BASE_URL=https://apihub.agnes-ai.com/v1
AI_MODEL=agnes-2.0-flash
# Speech-to-text
STT_MODE=tencent
WHISPER_MODEL=large-v3-turbo
# Tencent Cloud ASR (for STT_MODE=tencent)
TENCENT_SECRET_ID=
TENCENT_SECRET_KEY=
TENCENT_APP_ID=
# Long press sensitivity (ms)
LONG_PRESS_DURATION=800
# Force all requests through Agent
FORCE_AGENT=true
4. Download Whisper Model / 下载 Whisper 模型
For STT_MODE=local, download the model file:
- Path:
%APPDATA%/onhands3/data/whisper/ - Download ggml-large-v3-turbo.bin (~1.5GB)
- whisper-cli.exe auto-downloads from whisper.cpp releases
Manual placement:
%APPDATA%/onhands3/data/whisper/
├── Release/
│ └── whisper-cli.exe
└── ggml-large-v3-turbo.bin
Usage / 使用
Dev Mode / 开发模式
npm run dev
If launching from inside an Electron app (e.g. CherryStudio), run
unset ELECTRON_RUN_AS_NODEfirst.
Controls / 操作
- Long press (default 800ms) → start recording → release to transcribe
- Click input button → switch to text input mode
- Hold ESC 5s → force kill
- Test shortcuts:
Ctrl+Shift+1— simulate recordingCtrl+Shift+2— simulate transcriptionCtrl+Shift+3— simulate routingCtrl+Shift+4— simulate execution with streamingCtrl+Shift+0— hide window
Build / 构建
npm run build
npm run preview
Architecture / 项目架构
src/
├── main/ # Electron main process
│ ├── index.ts # Window creation, global shortcuts, IPC
│ ├── config.ts # Environment config loader
│ ├── orchestrator/
│ │ └── Orchestrator.ts # Core: voice → route → execute → result
│ ├── agents/
│ │ ├── ClaudeCodeAgent.ts # Claude Code CLI wrapper (stream-json)
│ │ └── AgentDetector.ts # Auto-detect installed Agent CLIs
│ ├── ai/
│ │ ├── DirectAI.ts # Direct AI API (quick tasks)
│ │ └── Router.ts # Command routing: direct vs agent
│ ├── input/
│ │ ├── MouseMonitor.ts # Mouse long-press detection (GetAsyncKeyState poll)
│ │ └── SelectionMonitor.ts # Text selection capture (child process + selection-hook)
│ ├── context/
│ │ └── ContextCollector.ts # Context: window, selected files/text, clipboard
│ ├── stt/
│ │ ├── WhisperSTT.ts # STT factory: local / cloud / tencent
│ │ └── TencentASR.ts # Tencent Cloud real-time ASR (WebSocket)
│ └── permission/
│ └── PermissionServer.ts # Dangerous operation approval system
├── renderer/ # Renderer process (React)
│ ├── App.tsx # UI state machine + glassmorphism overlay
│ ├── hooks/useVoiceRecorder.ts # Microphone recording (MediaRecorder API)
│ └── styles.css # All styles (no Tailwind)
├── preload/
│ └── index.ts # IPC bridge (contextBridge)
├── shared/
│ └── types.ts # Shared type definitions
scripts/
└── selection-worker.cjs # selection-hook worker (runs in child process)
Execution Flow / 执行流程
Long press → hide overlay → capture foreground window → show recording UI
↓
Release → audio data via IPC → Whisper STT transcription
↓
Route (Direct AI / Agent CLI)
↓
Collect context (window, selected text, files, clipboard, working dir)
↓
Execute → stream output → show result / auto-hide after 12s
Tech Stack / 技术栈
| Layer | Technology |
|---|---|
| Framework | Electron 35 + electron-vite 3 |
| UI | React 18 + TypeScript 5.7 |
| STT | Tencent Cloud ASR (streaming) / whisper.cpp (local fallback) / OpenAI Whisper API (cloud) |
| Agent | Claude Code CLI (stream-json mode) |
| Text selection | selection-hook via child process (UIA + IAccessible + Clipboard fallback) |
| Window capture | Win32 API via koffi (GetForegroundWindow, Shell COM) |
| Mouse input | Win32 API via koffi (GetAsyncKeyState polling) |
| Audio | ffmpeg-static (webm → wav conversion) |
Config Reference / 配置说明
| Variable | Description | Default |
|---|---|---|
AI_API_KEY | Direct AI API key | - |
AI_BASE_URL | AI API endpoint | https://apihub.agnes-ai.com/v1 |
AI_MODEL | AI model name | agnes-2.0-flash |
STT_MODE | STT mode: local / cloud / tencent | cloud |
WHISPER_MODEL | Whisper model size | large-v3-turbo |
LONG_PRESS_DURATION | Long press threshold (ms) | 800 |
DRAG_THRESHOLD_PX | Drag detection threshold (px) | 15 |
FORCE_AGENT | Force all requests to Agent | false |
Known Issues / 已知问题
- White title bar flash — Electron 35 bug (#47946), DWM briefly draws white chrome on focus change. Fixed in Electron 37.3.1+
- Agent encoding — Chinese path operations occasionally hit UTF-8 issues. Mitigated with single-quote wrapping + UTF-8 prefix
- Windows only — MouseMonitor uses Win32 API. macOS/Linux needs reimplementation
- GPU cache — If app exits immediately with GPU cache errors, delete
%LOCALAPPDATA%/OnHands3/GPUCacheand restart
Version History / 版本历史
| Version | Date | Summary |
|---|---|---|
| v0.48 | 2026-06-09 | Tencent Cloud streaming ASR, Phase 1 stability (history, parallel capture, crash recovery), permission dialog fix |
| v0.47 | 2026-06-08 | selection-hook replaces Ctrl+C for text selection capture |
| v0.46 | 2026-06-07 | ASK protocol — agent can present choices to user |
| v0.45 | 2026-06-07 | Permission API with proper Chinese encoding |
| v0.44 | 2026-06-06 | Permission server + Shell COM merge |
| v0.43 | 2026-06-06 | Fix foreground capture on re-trigger + img2img mode |
| v0.42 | 2026-06-05 | Universal media preview + selected image context |
Roadmap / 路线图
Phase 1 — Core Experience / 基础体验 ✅
Make it fast enough and stable enough that people want to use it daily.
| # | Item | Description | Status |
|---|---|---|---|
| 1.1 | Speed / 速度优化 | Parallel context collection, Tencent Cloud streaming STT, 5-min processing timeout | ✅ Done |
| 1.2 | Stability / 稳定性 | Agent crash recovery, STT auto-fallback (tencent→local), permission dialog fix, network error handling | ✅ Done |
| 1.3 | Recent History / 近期历史 | In-memory ring buffer (last 5 commands + result summaries), injected into prompt | ✅ Done |
Phase 2 — Context Intelligence / 上下文智能
Richer context → fewer ASK prompts → higher success rate per invocation.
| # | Item | Description |
|---|---|---|
| 2.1 | Timestamp / 时间上下文 | Current time, day of week, date. Handles "today", "tomorrow", "next Monday" |
| 2.2 | UIA Element Detection / 元素检测 | Detect element type at long-press position (image/link/button/input). Get src/href/alt text |
| 2.3 | Screen OCR / 屏幕文字识别 | OCR screenshot text into context directly. Saves one agent tool-call round |
| 2.4 | User Preferences / 用户偏好 | preferences.json: language, saveDir, commonProjects. Injected into prompt for consistent behavior |
Phase 3 — UI & Usability / 界面与可用性
Polish after the foundation is solid.
| # | Item | Description |
|---|---|---|
| 3.1 | Markdown Rendering / Markdown 渲染 | Headings, lists, code blocks (syntax highlight), tables, bold/italic |
| 3.2 | Result Layout / 结果排版 | Spacing, font sizing, long-text fold/scroll |
| 3.3 | Settings Page / 前端配置页 | GUI for API Key, STT mode, long-press duration. Replace manual .env editing |
Phase 4 — Ecosystem / 开放生态
Requires a user base to justify.
| # | Item | Description |
|---|---|---|
| 4.1 | SDK & Docs / SDK 文档 | Write after API freeze: IPC protocol, Agent protocol, ASK protocol |
| 4.2 | Module Marketplace / 模块市场 | Plugin API + frontend marketplace page |
Phases are soft boundaries — items can move across phases based on actual need.
Acknowledgments / 致谢
- Multica — AI desktop assistant framework
- selection-hook — cross-platform text selection detection
License
MIT