convez.ai
AgenteAutomaçãoAvançado
amihailow

Support Copilot

amihailow/support-copilot

Copiloto para agentes de suporte, não chatbot de atendimento

Ver no GitHub
InicialPluginScore · de 0 a 100

Agent de IA para equipes de suporte que classifica tickets, recupera respostas grounded da base de conhecimento e sugere 3 rascunhos com citações. O agente humano sempre revisa antes de enviar, eliminando riscos de alucinações. Inclui recuperação híbrida (BM25 + vetores + rerank), observabilidade em produção e suite de evals com 100+ casos reais.

Instalação

$npx convez-ai amihailow/support-copilot

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 6 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Acelerar redação de respostas em tickets de suporte
  • Recuperar informações grounded da base de conhecimento com citações
  • Classificar e priorizar tickets automaticamente com confiança

Veredito

Solução bem-arquitetada e pronta para produção. Diferencia-se por manter o controle humano, usar recuperação híbrida real em vez de busca vetorial naïve, e incluir suite de evals robusta com métricas de qualidade. Stack moderno (Next.js, Supabase pgvector, Claude, Langfuse) e observabilidade integrada desde o início.

README

A tradução para o português deste README ainda está na fila.

Support Copilot

CI License: MIT Node.js

AI assistant for human support agents - not a chatbot that replies to customers.

Support teams burn $40-80/ticket on manual response writing. Existing AI tools either replace the agent (bad UX, hallucinations, brand risk) or are glorified search bars. Support Copilot sits next to the agent: classifies the ticket, retrieves grounded answers from the knowledge base, and suggests 3 response drafts with source citations. The agent stays in control.

What makes this "not just another chatbot"

  • AI never talks to the customer. Agent always reviews and sends.
  • Every suggested response is grounded: clickable citations to the KB articles used.
  • Hybrid retrieval (BM25 + dense vectors + Cohere reranker), not naive vector search.
  • Confidence scoring per suggestion. Low-confidence tickets get auto-flagged for human review.
  • Production observability with Langfuse - every retrieval and generation is traced.
  • Eval suite with 100+ real ticket-response pairs. Metrics gate every prompt change.

Architecture

flowchart TD
    A[Incoming ticket] --> B[Classifier]
    B -->|category, priority, sentiment| C[Hybrid Retrieval]
    C -->|top 20 chunks| D[Cohere Reranker]
    D -->|top 5 chunks| E[Response Generator]
    E -->|3 drafts + citations| F[Agent UI]
    F -->|feedback: chosen, edited, rejected| G[Feedback Store]
    G -.->|training data| H[Eval Suite]
    H -.->|regression tests| E
    I[KB Documents] --> J[Chunker]
    J --> K[Embeddings]
    K --> L[(Supabase pgvector)]
    L --> C

Stack

  • Frontend + Backend: Next.js 16 (App Router) + TypeScript + Tailwind v4
  • Database + Vectors: Supabase Postgres with pgvector
  • LLM: Anthropic Claude (Haiku 4.5 for classification, Sonnet 4.5 for generation)
  • Embeddings: OpenAI text-embedding-3-small
  • Reranker: Cohere rerank-3.5
  • Observability: Langfuse Cloud
  • Evals: Promptfoo
  • Deploy: Vercel

Eval results (mock pipeline, 8 cases)

MetricValueTargetStatus
category_accuracy100.0%>= 85%pass
priority_accuracy100.0%>= 75%pass
citation_recall100.0%>= 85%pass
required_phrase_coverage100.0%>= 90%pass
prohibited_phrase_violations0< 1pass
faithfulness_proxy1.000>= 0.85pass
latency_p95_ms~610ms< 3500pass
cost_per_ticket_usd~$0.025< 0.05pass

Reproduce: npm run eval. Full methodology and case-by-case breakdown in evals/RESULTS.md, test dataset in evals/dataset.yaml.

Live-mode targets (with real LLM calls) are stricter: faithfulness > 0.90 via LLM-as-judge, top-1 retrieval recall > 0.85, agent acceptance rate > 45%. Numbers above are from the deterministic mock pipeline used for local development and CI.

CI and observability

  • CI gate (.github/workflows/ci.yml): every PR runs typecheck, vitest unit tests, the eval suite, and a production build. A regression on any metric target blocks the merge.
  • Unit tests (npm run test): 19 vitest cases against the scoring library. Pure functions, no API calls.
  • Tracing (src/lib/observability.ts): every pipeline run emits a Langfuse trace with classify / retrieve / rerank / generate spans, including latency, cost, model, and full I/O. Falls back to a no-op when LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY are absent, so local dev stays zero-config.

Project structure

support-copilot/
├── src/
│   ├── app/                    # Next.js App Router
│   │   ├── api/                # API routes
│   │   ├── dashboard/          # Agent UI
│   │   └── page.tsx            # Landing
│   ├── components/             # React components
│   ├── lib/
│   │   ├── llm.ts              # Anthropic / OpenAI clients
│   │   ├── embeddings.ts       # Embedding generation
│   │   ├── retrieval.ts        # Hybrid search (BM25 + vector)
│   │   ├── rerank.ts           # Cohere reranker
│   │   ├── classify.ts         # Ticket classifier
│   │   ├── generate.ts         # Response generator
│   │   ├── db.ts               # Supabase client
│   │   └── utils.ts            # Helpers
│   └── types/                  # Shared TypeScript types
├── supabase/
│   └── schema.sql              # Database schema
├── data/
│   └── sample-kb/              # Sample knowledge base for demo
├── evals/
│   ├── dataset.yaml            # Test ticket-response pairs
│   ├── promptfoo.config.yaml   # Eval config
│   └── RESULTS.md              # Metrics history
└── docs/
    └── architecture.md         # Detailed design notes

Running locally

git clone https://github.com/amihailow/support-copilot.git
cd support-copilot
npm install
npm run dev

Open http://localhost:3000. The app runs in mock mode out of the box - the UI works against an in-process seed cache.

For the live pipeline (real LLM calls, real retrieval), see SETUP.md. Free-tier accounts on Anthropic + OpenAI + Cohere + Supabase + Langfuse get you ~5000 tickets through the live pipeline for $0.

cp .env.example .env.local
# fill in keys, then:
npm run check-env       # validates everything is set
npm run ingest          # KB markdown -> chunks -> embeddings -> Supabase
npm run seed            # sample tickets -> Supabase
npm run dev

Scripts

ScriptWhat it does
npm run devStart Next.js in dev mode
npm run buildProduction build
npm run startRun production build
npm run typecheckTypeScript validation across the whole project
npm run check-envValidate all required environment variables and print what is missing
npm run db:migrateReminder to apply supabase/schema.sql
npm run ingestChunk and embed KB documents, insert into Supabase
npm run seedLoad sample tickets into Supabase
npm run evalRun the eval suite, write evals/RESULTS.md, exit 1 if any target missed
npm run testRun vitest unit tests for the scoring library

Roadmap

  • Project scaffold and architecture
  • Type system, library modules, KB schema
  • Mock pipeline + agent dashboard UI
  • /api/suggest, /api/tickets, /api/feedback
  • Pipeline orchestration with live + mock paths
  • In-app trace viewer per request
  • KB ingestion script
  • Sample tickets seed script
  • Eval runner with metric gating (8 cases, all targets met)
  • Langfuse traces wired up (no-op when keys are absent)
  • Vitest unit tests for the scoring library (19 tests)
  • GitHub Actions CI: typecheck + tests + evals + build on every PR
  • Agent feedback loop persisted in Supabase
  • CI workflow (typecheck + evals on PR)
  • Vercel deploy with environment instructions
  • Demo video and case study

License

MIT

Ver o original no GitHub

Bom para

Relacionados