convez.ai
AgenteAutomaçãoAvançado
Juadsuarezsan

Voice AI Agent Conversacional

Juadsuarezsan/voice-ai-agent

STT + LLM reasoning + TTS com <800ms por turno para telefonía IA

Ver no GitHub
InicialPluginScore · de 0 a 100

Agent conversacional por voz que combina Whisper (STT), Claude (raciocínio) e ElevenLabs (TTS) com latência sub-segundo por turno. Implementa padrão de preenchimento de slots e detecção de fim de fala com VAD. Pronto para casos como agendamento, atendimento ao cliente e comércio conversacional.

Instalação

$npx convez-ai Juadsuarezsan/voice-ai-agent

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 1 mês
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Atendimento ao cliente automatizado por voz
  • Agendamento de reservas (restaurante, medicina, etc)
  • Suporte de primeira linha conversacional
  • Comércio conversacional e vendas por voz

Veredito

Projeto maduro e bem-arquitetado que implementa corretamente o pipeline completo de voz (VAD → STT → reasoning → TTS). Roadmap claro com definição de sucesso mensurável (WER, latência, taxa de resolução). Recomendado para quem quer produzir voice agents em produção com raciocínio estruturado.

README

A tradução para o português deste README ainda está na fila.

Project 09 — Voice AI Conversational Agent

Telephony customer service over voice: Whisper STT → Claude reasoning → ElevenLabs TTS. Target under-800ms per turn for natural conversation. The pattern behind Bland AI, Vapi, Retell, Hume, Pindrop.

Status STT TTS

Industrial use case: Voice AI is one of the fastest-growing AI verticals — restaurant booking, medical scheduling, first-line customer support, conversational commerce.

What this project does

Closes the voice loop: caller speaks → VAD detects end of turn → Whisper transcribes → Claude reasons (with optional KB retrieval) → ElevenLabs synthesizes voice response → stream back to caller. Target latency under 800 ms end-to-end per turn for natural conversation.

Architecture

Audio input (WebRTC / phone / file)
   │
   ▼
[VAD] Silero VAD detects turn end
   │
   ▼
[STT] Whisper-large-v3
   │ → text + confidence + detected language
   │
   ▼
[Intent + Context] LangGraph state
   │ ├─ conversation history
   │ ├─ slots completed (date, party_size, etc.)
   │ └─ current task type
   │
   ▼
[Reasoner] Claude → next action: respond | ask_clarification | book | transfer_to_human
   ├──► if needs info: RAG over KB
   └──► if needs action: tool call (booking API, etc.)
   │
   ▼
[Response Generator] Claude → natural conversational text
   │
   ▼
[TTS] ElevenLabs or XTTS-v2 → streamed audio
   │
   ▼
[Audio Stream] back to caller
   │
   ▼
[Conversation Logger] PostgreSQL — transcript, audio refs, latencies per turn

Roadmap to v1.0.0

  1. Whisper-large-v3 local setup (GPU) or Whisper API integration
  2. Silero VAD for end-of-turn detection
  3. LangGraph state with conversation history + slots
  4. Claude reasoning with structured output (action + parameters)
  5. Tool integrations (mock booking API for demo)
  6. ElevenLabs TTS with streaming
  7. XTTS-v2 local fallback for cost-free demos
  8. WER eval on LibriSpeech test-clean + test-other + Spanish Common Voice
  9. Resolution rate eval over 100 simulated conversations (LLM-as-judge)
  10. Next.js demo with WebRTC for browser audio
  11. 10 recorded conversations (MP3 in repo) of real task completions

Stack

LayerTechnology
STTopenai-whisper whisper-large-v3 (local, GPU) or Whisper API ($0.006/min)
LLMClaude Sonnet 4.5
TTSElevenLabs API ($0.30/1K chars) or coqui-tts XTTS-v2 (local)
OrchestrationLangGraph with turn detection
StreamingLiveKit Cloud or Twilio Voice for production
VADsilero-vad
StoragePostgreSQL for transcripts, S3/filesystem for audio
FrontendNext.js + WebRTC for browser audio
ObservabilityLangSmith + custom per-component latency metrics

Definition of Done — project-specific

  • WER reported on at least 2 benchmarks (LibriSpeech + Spanish dataset)
  • End-to-end per-turn latency measured + optimized under 1 second
  • 100 simulated conversations evaluated with LLM-as-judge for resolution rate
  • Demo: WebRTC interface — speak in browser, receive voice response
  • 10 real recorded conversations (MP3) included in repo as evidence
  • Whisper open vs API comparison with costs documented
  • XTTS-v2 vs ElevenLabs comparison with MOS documented
  • Demo conversation lasts ≥6 turns completing a real task

Plus the 12 universal DoD blocks.

License

MIT.

Ver o original no GitHub

Bom para

Relacionados