Voice AI Agent Conversacional
Juadsuarezsan/voice-ai-agent
STT + LLM reasoning + TTS com <800ms por turno para telefonía IA
Ver no GitHubAgent conversacional por voz que combina Whisper (STT), Claude (raciocínio) e ElevenLabs (TTS) com latência sub-segundo por turno. Implementa padrão de preenchimento de slots e detecção de fim de fala com VAD. Pronto para casos como agendamento, atendimento ao cliente e comércio conversacional.
Instalação
npx convez-ai Juadsuarezsan/voice-ai-agent1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Atendimento ao cliente automatizado por voz
- Agendamento de reservas (restaurante, medicina, etc)
- Suporte de primeira linha conversacional
- Comércio conversacional e vendas por voz
Veredito

Projeto maduro e bem-arquitetado que implementa corretamente o pipeline completo de voz (VAD → STT → reasoning → TTS). Roadmap claro com definição de sucesso mensurável (WER, latência, taxa de resolução). Recomendado para quem quer produzir voice agents em produção com raciocínio estruturado.
README
A tradução para o português deste README ainda está na fila.
Project 09 — Voice AI Conversational Agent
Telephony customer service over voice: Whisper STT → Claude reasoning → ElevenLabs TTS. Target under-800ms per turn for natural conversation. The pattern behind Bland AI, Vapi, Retell, Hume, Pindrop.
Industrial use case: Voice AI is one of the fastest-growing AI verticals — restaurant booking, medical scheduling, first-line customer support, conversational commerce.
What this project does
Closes the voice loop: caller speaks → VAD detects end of turn → Whisper transcribes → Claude reasons (with optional KB retrieval) → ElevenLabs synthesizes voice response → stream back to caller. Target latency under 800 ms end-to-end per turn for natural conversation.
Architecture
Audio input (WebRTC / phone / file)
│
▼
[VAD] Silero VAD detects turn end
│
▼
[STT] Whisper-large-v3
│ → text + confidence + detected language
│
▼
[Intent + Context] LangGraph state
│ ├─ conversation history
│ ├─ slots completed (date, party_size, etc.)
│ └─ current task type
│
▼
[Reasoner] Claude → next action: respond | ask_clarification | book | transfer_to_human
├──► if needs info: RAG over KB
└──► if needs action: tool call (booking API, etc.)
│
▼
[Response Generator] Claude → natural conversational text
│
▼
[TTS] ElevenLabs or XTTS-v2 → streamed audio
│
▼
[Audio Stream] back to caller
│
▼
[Conversation Logger] PostgreSQL — transcript, audio refs, latencies per turn
Roadmap to v1.0.0
- Whisper-large-v3 local setup (GPU) or Whisper API integration
- Silero VAD for end-of-turn detection
- LangGraph state with conversation history + slots
- Claude reasoning with structured output (action + parameters)
- Tool integrations (mock booking API for demo)
- ElevenLabs TTS with streaming
- XTTS-v2 local fallback for cost-free demos
- WER eval on LibriSpeech test-clean + test-other + Spanish Common Voice
- Resolution rate eval over 100 simulated conversations (LLM-as-judge)
- Next.js demo with WebRTC for browser audio
- 10 recorded conversations (MP3 in repo) of real task completions
Stack
| Layer | Technology |
|---|---|
| STT | openai-whisper whisper-large-v3 (local, GPU) or Whisper API ($0.006/min) |
| LLM | Claude Sonnet 4.5 |
| TTS | ElevenLabs API ($0.30/1K chars) or coqui-tts XTTS-v2 (local) |
| Orchestration | LangGraph with turn detection |
| Streaming | LiveKit Cloud or Twilio Voice for production |
| VAD | silero-vad |
| Storage | PostgreSQL for transcripts, S3/filesystem for audio |
| Frontend | Next.js + WebRTC for browser audio |
| Observability | LangSmith + custom per-component latency metrics |
Definition of Done — project-specific
- WER reported on at least 2 benchmarks (LibriSpeech + Spanish dataset)
- End-to-end per-turn latency measured + optimized under 1 second
- 100 simulated conversations evaluated with LLM-as-judge for resolution rate
- Demo: WebRTC interface — speak in browser, receive voice response
- 10 real recorded conversations (MP3) included in repo as evidence
- Whisper open vs API comparison with costs documented
- XTTS-v2 vs ElevenLabs comparison with MOS documented
- Demo conversation lasts ≥6 turns completing a real task
Plus the 12 universal DoD blocks.
License
MIT.