convez.ai
MCP ServerDados & AnalyticsIntermediário
Michaela2024

Product Analytics Agent

Michaela2024/product-analytics-agent

Agente IA para análise de produto conversacional com detecção de anomalias

Ver no GitHub
InicialPluginScore · de 0 a 100

MCP server que implementa um agente de IA conversacional para responder perguntas sobre métricas de produto sem escrever SQL. O agente consulta dados, compara segmentos, detecta anomalias e gera gráficos automaticamente através de chamadas de ferramentas estruturadas.

Instalação

$npx convez-ai Michaela2024/product-analytics-agent

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
28%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Investigar quedas de métricas e anomalias de produto
  • Comparar comportamento entre segmentos de usuários
  • Gerar insights e visualizações sem SQL

Veredito

Implementação sólida de agente MCP com tool use responsável (SQL encapsulado vs. geração). Usa Haiku para eficiência de custo e padrões realistas (dados sintéticos com anomalias seeded). Adequado para product managers que querem análise conversacional segura e auditável.

README

A tradução para o português deste README ainda está na fila.

Product Analytics Agent

An AI-powered conversational analytics tool that lets product leaders query metrics, compare segments, detect anomalies, and visualise trends — without writing SQL.

The Problem

Product leaders spend hours context-switching between dashboards, SQL editors, and spreadsheets to answer basic questions about their product. "What happened to activation this week?" shouldn't require writing a query, building a chart, and interpreting the results manually. It should be a conversation.

What This Does

A product manager types a natural language question. The AI agent autonomously decides which analytical tools to call, queries the database, interprets the results, and responds with insights and charts — all in a chat interface.

Example interaction:

User: "What happened to activation rates recently?"

Agent: Calls detect_anomalies(metric='activation_rate', lookback_days=14) → finds a 20.5 percentage point drop → explains the finding and suggests next steps.

The agent handles multi-step reasoning: it can detect an anomaly, then compare segments to see if it's concentrated, then generate a chart — all from a single question.

Architecture

User Question
     ↓
Streamlit Frontend (chat UI)
     ↓
Agent (Anthropic SDK + Claude Haiku 4.5)
     ↓                    ↑
Tool Selection ←→ Tool Results
     ↓
MCP Server (FastMCP)
     ↓
SQLite Database (synthetic product data)

Key components:

  • Frontend (app.py): Streamlit chat interface with a sidebar showing the agent's tool calls for transparency. Example prompts help new users get started.
  • Agent (agent.py): Manages the conversation loop. Sends the user's question to Claude along with tool definitions. When Claude decides to call a tool, the agent executes it and feeds the result back. This loop continues until Claude has enough data to give a final answer.
  • MCP Server (mcp_server.py): Exposes four analytical tools via FastMCP: query_metrics, compare_segments, detect_anomalies, and get_feature_usage. Each tool runs SQL against the database and returns structured results.
  • Data Layer (generate_data.py): Generates ~95,000 rows of synthetic product analytics data across four tables (users, events, funnel_steps, subscriptions) with deliberate patterns seeded in.

Design Decisions

Why synthetic data with seeded patterns? Real product data is confidential and messy. Synthetic data lets me control the narrative: I seeded three specific patterns (an activation drop, a segment retention gap, and accelerating feature adoption) so the agent always has something meaningful to find. This also makes the eval suite deterministic — I can verify the agent finds the right patterns every time.

Why Claude Haiku over a larger model? Cost-efficiency. Haiku at $1/$5 per million tokens handles tool selection accurately for this use case. During development and testing, total API spend was under $2. In production, you'd monitor per-query costs and potentially route complex questions to a larger model.

Why tool use over raw SQL generation? Having the LLM write arbitrary SQL against a production database is a trust and safety problem. Instead, the agent selects from predefined, tested tools — each tool encapsulates a validated query pattern. This is safer, more predictable, and easier to audit.

Why show tool calls in the sidebar? Transparency. A product leader needs to trust the numbers. Showing which tools were called and with what parameters lets users verify the agent's reasoning, not just its conclusions. This matters in production where decisions have consequences.

Eval Results

The test suite (test_suite.py) covers three layers:

Data quality (6 tests): No future dates, correct row counts, all segments and statuses present.

Seeded patterns (3 tests): Enterprise churn is lower than SMB, activation dropped in the last two weeks, AI assistant adoption accelerated after March 2026.

Tool correctness (5 tests): Each tool returns expected output, chart data is valid JSON with no future months.

Agent behaviour (3 tests): The agent selects detect_anomalies for anomaly questions, compare_segments for comparison questions, and generate_chart for visualisation requests.

Result: 20/20 tests passing.

The initial suite passed 18/18 but missed an edge case: functions called without optional parameters failed because the MCP framework's decorators injected validation objects into the default values. I discovered this during integration testing when the LLM omitted optional parameters, added two tests for default parameter handling, and resolved it by separating the core analytics logic from the MCP decorator layer.

Known Limitations

  • Synthetic data only. The patterns are seeded, not discovered. Real-world data would surface messier, more ambiguous signals.
  • No authentication. Any user can query any data. Production would need role-based access control and data-level permissions.
  • No conversation persistence. Chat history is lost when the session ends. Product leaders would want saved queries and scheduled reports.
  • Case-sensitive segments. Users must type "smb" not "SMB". Production would normalise inputs.
  • Single model. All queries go to Haiku. A production system might route complex analytical questions to a larger model while keeping simple lookups on Haiku.
  • No cost monitoring. Each query makes API calls with no tracking. Production needs per-query cost logging and budget alerts.

What Production Would Look Like

  • Real data sources: Replace SQLite with a data warehouse (BigQuery, Snowflake) or a metrics layer (Looker, dbt). The MCP tool interface stays the same — only the query implementations change.
  • Authentication and multi-tenancy: OAuth login, team-scoped data access, audit logging of every query and tool call.
  • Scheduled reports: A cron job that runs predefined questions weekly and sends summaries to Slack or email — same agent code, different trigger.
  • Cost controls: Per-query token tracking, budget limits per team, model routing based on query complexity.
  • Query validation: A review layer that checks generated insights against known baselines before surfacing them to users.

Tech Stack

  • Python 3.11
  • Anthropic SDK (Claude Haiku 4.5)
  • FastMCP (Model Context Protocol server)
  • SQLite (synthetic data store)
  • Streamlit (frontend)
  • Pydantic (tool parameter validation)

Running Locally

# Clone and setup
git clone [repo-url]
cd product-analytics-agent
python -m venv venv
source venv/bin/activate  # Windows: .\venv\Scripts\Activate.ps1

# Install dependencies
pip install anthropic python-dotenv mcp streamlit pandas

# Configure
echo "ANTHROPIC_API_KEY=your-key-here" > .env

# Generate data
python generate_data.py

# Run tests
python test_suite.py

# Start the app
streamlit run app.py

What I Learned

This project was built to understand AI agent architecture from first-hand experience — not from reading about it, but from building it. Key takeaways:

  • Tool design is product design. How you split functionality into tools determines what questions the agent can answer. Get the tool boundaries wrong and the agent either can't answer the question or calls five tools when one would do.
  • The LLM is surprisingly good at tool selection. With clear descriptions, Claude consistently picks the right tool. The descriptions matter more than the code — a poorly described tool gets ignored even if it's the right one.
  • Evals are essential, not optional. The agent's tool selection is non-deterministic. Without automated tests, you're manually checking every interaction and hoping for the best.
  • The hard part isn't the protocol. MCP, tool decorators, and the API are straightforward. The hard part is data quality, trust, and knowing which questions are worth answering — which is a product problem, not an engineering one.
Ver o original no GitHub

Bom para

Relacionados