convez.ai
AgenteProdutividadeAvançado
TaylorFinklea

Hermes Voice

TaylorFinklea/hermes-voice

Interface iOS por voz para agentes de codificação locais com backend FastAPI

Ver no GitHub
InicialPluginScore · de 0 a 100

Agent iOS nativo para voz que conecta um iPhone a um agente de codificação local (Hermes, Claude Code, etc.) em um Mac, via backend FastAPI sobre Tailscale. Funciona com push-to-talk, reconhecimento de fala, execução de código e síntese de voz em ciclos completos.

Instalação

$npx convez-ai TaylorFinklea/hermes-voice

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
2
Forks
1
Acompanhando
0
Saúde
42%
Último commit
há 3 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Executar comandos de codificação no Mac via voz do iPhone
  • Manter estado de ferramentas e contexto em múltiplas rodadas de interação
  • Substituir interfaces de chat de voz precárias com aplicativo dedicado

Veredito

Projeto técnico robusto com arquitetura bem definida (STT → LLM → TTS), suporte a múltiplos provedores e provisionamento seguro via Tailscale. Ideal para quem quer interface dedicada por voz para agentes locais, mas exige infraestrutura macOS e conhecimento DevOps.

README

A tradução para o português deste README ainda está na fila.

Harness Voice

Voice-native iPhone front-end for a local coding agent (Hermes, Claude Code, Codex, or OpenCode). Push-to-talk on the phone, Hermes does the actual operating on the Mac, audio answer plays back. Built to replace the rough Telegram-voice experience with something dedicated.

Architecture

┌──────────────┐   audio (m4a)   ┌──────────────────┐   subprocess   ┌───────────┐
│ iOS app      │ ──────────────▶ │ FastAPI backend  │ ─────────────▶ │ hermes CLI│
│ (SwiftUI)    │                 │  /api/audio      │                │ on macOS  │
│              │ ◀────────────── │  /api/text       │ ◀───────────── │           │
└──────────────┘  TurnResponse   │  STT + TTS layer │   stdout/      └───────────┘
                  (text + URL)   └──────────────────┘   session_id
                                          │
                       ┌──────────────────┼──────────────────┐
                       ▼                  ▼                  ▼
                 STT providers       TTS providers      Audio cache
                 (OpenAI, Groq,      (ElevenLabs,       (tmp dir,
                  faster-whisper)    OpenAI, Piper)     served by id)

A turn is one round trip: iPhone records → POSTs audio → backend STT → hermes chat -Q -q <text> (with --resume <id> on follow-ups so Hermes keeps tool/context state) → backend TTS → iPhone plays mp3/wav.

What's in the box

  • backend/ — FastAPI service. Provider abstractions for STT (OpenAI / Groq / local faster-whisper / mock) and TTS (ElevenLabs / OpenAI / local Piper / mock). 18 unit tests with fake providers.
  • ios/HermesVoice/ — SwiftUI app, regenerable via XcodeGen from project.yml. Push-to-talk that supports both hold-to-talk and tap-to-latch. Always-visible text input fallback.

Quick start

1. Set up a Mac server (recommended)

From the app's Set up a Mac server screen, copy the versioned prompt into a local coding harness running on the target Mac. The prompt runs the first-party provisioner and pauses before it configures a persistent tailnet-only HTTPS proxy (Tailscale Serve) and installs one backend LaunchAgent. It never asks you to paste a token into the chat, and it never generates or requests a certificate file.

The provisioner supports these commands. Run them from the repository root—the checkout directory containing backend/:

uv run --project backend --extra provision python backend/scripts/provision_macos.py setup
uv run --project backend --extra provision python backend/scripts/provision_macos.py setup --from-bws
uv run --project backend --extra provision python backend/scripts/provision_macos.py pair

setup discovers the checkout, uv, Tailscale, and MagicDNS name; binds the backend to loopback; reuses or adds the exact Tailscale Serve HTTPS mapping to 127.0.0.1:8765; stores the server token in the macOS Keychain; installs one io.harnessvoice.backend LaunchAgent; and checks the public HTTPS /health endpoint before reporting ready. It never calls tailscale serve reset and never touches an unrelated Serve mapping — a conflicting existing mapping on port 8765 stops setup before any local change. --from-bws is an opt-in path for a registered Bitwarden Secrets Manager project; the managed token is passed to a child process through its environment only. If any step after the mapping/LaunchAgent cutover fails (public health, authenticated harness check), setup automatically rolls back everything it changed and restores what was previously running; rerunning setup after fixing the underlying problem is always safe.

pair shows a five-minute, one-time QR code, but only once every readiness gate passes: a server token already stored in Keychain, an exact non-conflicting Tailscale Serve mapping (or, for the advanced manual direct-TLS runtime, both HERMES_VOICE_CERT and HERMES_VOICE_KEY set), a public HTTPS /health that reports mock: false, and an authenticated /api/harnesses listing showing Hermes available. Scan the code from the app; the long-lived server token is never encoded in the QR code, and the pairing secret is never printed separately from the code itself.

Troubleshooting guided setup

Guided setup supports macOS with Tailscale only. Run it from a local coding harness on the target Mac and approve the Serve/LaunchAgent configuration when prompted. If setup stops after a prerequisite changes, rerun setup; to replace an expired or used pairing code, rerun pair. If camera access is unavailable, keep using the app's manual URL/token or Bonjour paths.

Non-secret, non-destructive diagnostics:

tailscale serve status --json
launchctl print "gui/$(id -u)/io.harnessvoice.backend"
tail -F "$HOME/Library/Logs/Harness Voice/backend.log"
curl -s "https://$(tailscale status --json | jq -r '.Self.DNSName | rtrimstr(\".\")'):8765/health" | jq

tailscale serve status --json only reads Serve state — it never mutates anything and is always safe to run before or after setup/pair. The health endpoint should report status: ok, scheme: https, and mock: false. Never print the Keychain token or paste credentials into a harness prompt, diagnostic command, issue, or chat. The copied app prompt contains only the public repository and provisioner command.

Advanced local development

For local-only development without a public Tailscale server:

cd backend
uv sync --extra dev
cp .env.example .env          # optional; do not use this for guided setup
uv run python -m app          # serves on http://127.0.0.1:8765

With no env vars set, the backend auto-engages mock mode. Smoke-test it with:

curl -s http://127.0.0.1:8765/health | jq
curl -s -X POST http://127.0.0.1:8765/api/text \
  -H 'content-type: application/json' \
  -d '{"text":"hello hermes"}' | jq

2. iOS app

cd ios/HermesVoice
xcodegen generate            # regenerates HermesVoice.xcodeproj from project.yml
open HermesVoice.xcodeproj

Then in Xcode:

  1. Select a development team in Signing & Capabilities (your personal team is fine).
  2. Pick a simulator (or your iPhone) and Run.
  3. First launch on a real phone: tap the gear icon and set the backend URL to your Mac's reachable address (Tailscale IP recommended, e.g. http://100.64.x.y:8765).
  4. Tap Ping /health in Settings to verify connectivity.

3. Reach your Mac from your phone

You have three options, in order of preference:

  1. Tailscale (best). Install Tailscale on both, run the backend with HERMES_VOICE_HOST=0.0.0.0, use your Mac's Tailscale IP in the iOS Settings screen.
  2. LAN: same Wi-Fi, find your Mac's local IP (ipconfig getifaddr en0), bind to 0.0.0.0.
  3. Simulator: leave host as 127.0.0.1 — no extra config.

⚠️ When binding to 0.0.0.0, set HERMES_VOICE_TOKEN=<random> and put the same token in the iOS Settings auth field. The token is required on every request when set.

Env vars (see backend/.env.example)

VariableDefaultWhat it does
HERMES_VOICE_HOST127.0.0.1Bind address (use 0.0.0.0 over Tailscale/LAN only).
HERMES_VOICE_PORT8765Listen port.
HERMES_VOICE_MOCKunsetForce mock mode even with keys present.
HERMES_VOICE_TOKENunsetIf set, requests must send X-Hermes-Voice-Token.
HERMES_BINhermesPath/name of the Hermes CLI.
HERMES_EXTRA_ARGSunsetAppended to every hermes chat (e.g. -t apple_notes,apple_reminders -m anthropic/claude-sonnet-4).
HERMES_TIMEOUT_SECONDS180Per-turn timeout.
STT_PROVIDERautoopenai / groq / elevenlabs / local_whisper / mock.
OPENAI_API_KEY / VOICE_TOOLS_OPENAI_KEYunsetUsed for OpenAI Whisper + OpenAI TTS.
GROQ_API_KEYunsetGroq Whisper-large-v3-turbo (very fast).
LOCAL_WHISPER_MODELbase.enfaster-whisper model name.
TTS_PROVIDERautoelevenlabs / openai / local_piper / mock.
ELEVENLABS_API_KEYunsetElevenLabs key (powers TTS and Scribe STT).
ELEVENLABS_VOICE_IDnPczCjzI2devNBz1zQrbBrian (deeper narrator).
OPENAI_TTS_MODELtts-1tts-1, tts-1-hd, or gpt-4o-mini-tts.
OPENAI_TTS_VOICEonyxMasculine fallback voice.
PIPER_VOICE_PATHunsetAbsolute path to a .onnx voice (with .onnx.json next to it).

Auto-starting the backend

The guided setup command generates one portable, user-specific io.harnessvoice.backend LaunchAgent under the current user's Library directories and starts it only after confirmation. It binds the backend to loopback and reuses Tailscale Serve for the tailnet-facing HTTPS proxy; it never generates a certificate-renewal agent, because Tailscale Serve renews its own certificate transparently. From the repository root—the checkout directory containing backend/—run:

uv run --project backend --extra provision python backend/scripts/provision_macos.py setup

The checked-in files under backend/launchd/ are deprecation stubs and must not be copied into ~/Library/LaunchAgents/. For advanced manual development, continue using the loopback command in the section above; do not adapt the checked-in stubs into a public service.

The iOS app's Settings → Diagnostics screen shows "Backend last seen" so you can spot the moment it goes unreachable.

Latency tuning

The slowest part of any turn is Hermes' LLM thinking + tool-call round-trips (2–6s typically). Beyond that, two knobs are worth knowing:

  • STT provider: Groq's Whisper-large-v3-turbo is noticeably faster than OpenAI's whisper-1 — often 200–400ms vs 600–1000ms for short utterances. Set STT_PROVIDER=groq and GROQ_API_KEY=... in .env to A/B.
  • TTS first-byte: ElevenLabs eleven_turbo_v2_5 delivers first audio in ~300ms. The _run_turn flow kicks off TTS streaming before the tool-call audit completes, so audit time doesn't push perceived audio start.

A/B-ing STT providers

If you have multiple keys set, the backend auto-picks openai first. To force a comparison without removing keys:

# in backend/.env
STT_PROVIDER=elevenlabs   # Scribe v1 — strong WER, good with accents
# STT_PROVIDER=groq       # whisper-large-v3-turbo — fastest by wall clock
# STT_PROVIDER=openai     # whisper-1 — the safe default
# STT_PROVIDER=local_whisper  # offline, requires '.[local]'

Restart the backend and call /health to confirm: the stt.name field reports which provider is live.

Scribe v2 streaming (the WebSocket "Scribe Realtime" product) is the upgrade path when we add streaming STT — it slots into the same STTProvider Protocol.

Fully local mode (no API calls)

cd backend
uv sync --extra dev --extra local
# Download a Piper voice (one-time, ~60MB):
mkdir -p ~/.local/share/piper-voices && cd ~/.local/share/piper-voices
curl -L -o en_US-ryan-high.onnx \
  https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/ryan/high/en_US-ryan-high.onnx
curl -L -o en_US-ryan-high.onnx.json \
  https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/ryan/high/en_US-ryan-high.onnx.json

Then in .env:

STT_PROVIDER=local_whisper
TTS_PROVIDER=local_piper
PIPER_VOICE_PATH=/Users/you/.local/share/piper-voices/en_US-ryan-high.onnx

Notes:

  • faster-whisper runs on CPU with int8 quantization on Apple Silicon — fine for short utterances. (Metal is not yet supported by CTranslate2.)
  • Parakeet would be the upgrade path for STT but its Apple Silicon story is still rough. There's a # TODO marker in backend/app/stt/local_whisper.py where the swap-in would go.
  • First Whisper transcription downloads the model (~150MB for base.en).

Tests

cd backend
uv run pytest -q

Provider abstractions are unit-testable without network: tests inject FakeHermes and FakeTTS directly into the app factory.

Current limitations

  • No streaming. v1 records the full utterance, sends after stop, gets one complete response, plays it. Real-time partial transcripts and streaming TTS are roadmap items.
  • No interruption / barge-in. While Hermes is speaking, the mic button is disabled.
  • No wake word / VAD. Push-to-talk only.
  • Session continuity is best-effort. The backend stores session_id round-trips, but if Hermes wipes a session, the next turn starts fresh.
  • No approval cards. Hermes runs whatever tools you have configured. Use HERMES_EXTRA_ARGS to restrict toolsets (e.g. -t apple_notes,apple_reminders for read-mostly).

Roadmap

  • WebSocket transport for the iOS ↔ backend leg (replace HTTP POSTs).
  • Streaming STT — partial transcripts visible in the transcript as you speak.
  • ElevenLabs streaming TTS — playback starts before synthesis finishes.
  • Interruption / barge-in — tapping the mic while Hermes speaks cuts him off and starts recording.
  • Approval cards — backend surfaces "Hermes wants to send iMessage to Dad: ...", iOS shows an Approve/Deny card before the action runs.
  • iOS Shortcut: "Hey Siri, ask Hermes ..." → fires /api/text and speaks the reply.
  • Home Assistant voice-satellite integration — share the same backend with HA Assist.
  • Background audio session improvements (currently playback uses .duckOthers).
  • Swap faster-whisper for NVIDIA Parakeet (parakeet-mlx) once it's plug-and-play on Apple Silicon.

Layout

hermes-voice/
├── backend/
│   ├── app/
│   │   ├── main.py            # FastAPI app + route registration
│   │   ├── config.py          # env-driven Settings
│   │   ├── models.py          # request/response shapes
│   │   ├── hermes.py          # subprocess-based HermesClient + MockHermesClient
│   │   ├── audio_store.py     # short-lived synthesized-audio cache
│   │   ├── stt/               # provider Protocol + openai/groq/local/mock impls
│   │   └── tts/               # provider Protocol + elevenlabs/openai/piper/mock impls
│   ├── tests/                 # pytest (no network, all-fake providers)
│   ├── pyproject.toml
│   └── .env.example
├── ios/HermesVoice/
│   ├── project.yml            # XcodeGen project spec
│   └── HermesVoice/
│       ├── HermesVoiceApp.swift
│       ├── Models/            # Message, AppSettings
│       ├── Services/          # VoiceRecorder, AudioPlayer, HermesVoiceAPI
│       ├── ViewModels/        # ConversationViewModel (state machine)
│       └── Views/             # MainView, TranscriptView, PushToTalkButton, SettingsView
└── README.md
Ver o original no GitHub

Bom para

Relacionados