convez.ai
AgenteDevOps & InfraestruturaIntermediário
ccf

Primer

ccf/primer

Medição de efetividade real de agents de IA para times de engenharia

Ver no GitHub
InicialPluginScore · de 0 a 100

Plataforma open-source de inteligência para agent harness que captura e analisa dados de sessão de ferramentas de IA (Claude Code, Codex CLI, Gemini CLI, Cursor) para medir efetividade real e guiar otimização de desenvolvimento. Oferece métricas de ROI, análise por time e scoring de qualidade com dados mantidos em sua rede privada.

Instalação

$npx convez-ai ccf/primer

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
57%
Último commit
há 4 meses
Versão
v0.2.0
Issues abertas
5
Contribuidores
3

Casos de uso

  • Medir ROI e custo por outcome bem-sucedido de ferramentas de IA coding
  • Identificar quais times dominam agent patterns e quais precisam de coaching
  • Otimizar estratégia de caching, context management e orquestração de agents
  • Comparar efetividade de PRs assistidos por IA versus código manual
  • Rastrear onboarding e ramping de novos engineers com IA

Veredito

Ferramenta madura e bem-engineered (92% cobertura, Python 3.12+) que resolve um problema real: times precisam dados concretos para otimizar uso de agents, não apenas intuição. Abordagem privacy-first e self-hosted é diferencial forte para empresas com compliance.

README

A tradução para o português deste README ainda está na fila.



Your team adopted AI coding tools. Now you need to know what to standardize.

Primer is an open-source agent harness intelligence platform for engineering teams. It captures session data from Claude Code, Codex CLI, Gemini CLI, and Cursor, then measures how your agent harness — tool design, context management, caching, orchestration, and permission boundaries — determines outcome quality. It turns that telemetry into harness effectiveness scores, quality attribution, cost proof, exemplar sessions, coaching, and experiments. Self-hosted. Privacy-first. Your data never leaves your network.

Questions Primer Answers

For engineering leadership

  • Is our AI investment paying off? — Cost per successful outcome, ROI ratio, time saved vs. dollars spent
  • Which teams have figured it out and which are struggling? — Team-level adoption, success rates, and leverage scores with cross-team comparison
  • What systemic issues tank AI effectiveness across the org? — Friction impact scoring: not just "errors happen" but "this error type costs us 40% lower success rates"
  • Are we spending efficiently? — Per-engineer plan modeling (API vs. Pro vs. Max), cache hit optimization, budget burn-rate tracking

For team leads

  • Who needs help and what specific help? — Personalized tips, skill gaps, tool diversity analysis, and peer benchmarking
  • How fast are new hires ramping up? — Cohort analysis comparing onboarding curves against team baselines
  • Are AI-assisted PRs actually better? — Side-by-side comparison: merge time, review comments, merge rate for Claude vs. non-Claude PRs
  • Where should I invest in training? — Tool proficiency scoring (novice → expert) per engineer, per tool category

For individual engineers

  • How am I trending? — Weekly trajectory with success rate, cost, and session patterns
  • What keeps tripping me up? — Personal friction breakdown: permission blocks, context limits, tool errors
  • Am I using the right tools? — Config optimization suggestions based on team benchmarks and entropy-based diversity scoring
  • What should I try next? — Learning paths generated from high-performer patterns in your org

What Makes This Different

Harness intelligence, not just usage tracking. Primer measures how your agent harness configuration — tool design, caching strategy, orchestration depth, context management, and permission boundaries — determines outcome quality. It links harness patterns to merge rates, review quality, cost, and reusable assets so you can decide which configurations to standardize.

Friction analysis, not just error counting. Primer doesn't just count failures. It classifies why sessions fail, what engineers tried next, and which recoveries actually worked. You learn that permission_denied errors cause lower success rates, not just that they happened sometimes.

Customization and orchestration intelligence. Primer captures MCPs, skills, commands, subagents, delegation patterns, and agent-team modes so you can see which explicit agent stacks differentiate high performers and which ones create friction.

Individual intelligence, not just org dashboards. Every engineer gets a trajectory view, strengths profile, friction breakdown, AI-generated narrative, model coach, and learning paths. The MCP sidecar now brings session-start coaching, live signals, in-session nudges, stats, friction reports, recaps, and manager review packs into the session without forcing context switching.

Cost optimization, not just cost tracking. Primer models whether each engineer should be on API billing, Pro ($20/mo), or Max ($100/mo) based on actual usage. It measures cache hit rates per engineer and surfaces how much money is being left on the table. Budget burn-rate alerts catch overruns before they happen.

Experiments, not just recommendations. Primer turns recommendations into interventions and structured experiments so you can measure whether a rollout, training change, or playbook actually improved outcomes.

Capabilities

AreaWhat You Get
Organization OverviewSession volume, success rates, health scores, activity heatmaps, outcome distribution, anomaly alerts
FinOps & Cost ManagementPer-model spend, cache savings analysis, API vs. subscription modeling, 30-day linear regression forecasting, budget burn-rate tracking
Engineer ProfilesWeekly trajectory sparklines, strengths/friction breakdown, peer benchmarking, AI-generated narrative insights
AI MaturityLeverage scores (tool mastery, orchestration depth, efficiency), Effectiveness scores (success rate, cost efficiency), model diversity, agent team detection, project AI-readiness
Friction IntelligenceCategorized friction types with impact scoring, bottleneck detection, root cause patterns, cluster analysis
Quality & Code ImpactPR merge rates, workflow-based quality attribution, post-merge outcomes, Claude-assisted vs. non-Claude comparison, code volume per session, review comment analysis
AI SynthesisLLM-generated narrative reports at org, team, and engineer scope — turns metrics into stories
Conversational ExplorerNatural language queries over your data via SSE-streamed tool-use chat
Session BrowserFull-text search, outcome/model/type filters, transcript viewer, message-level detail
MCP SidecarSession-start coaching, live signals, in-session nudges, personal recaps, manager review packs, and mid-session stats
Multi-Agent SupportClaude Code, Codex CLI, Gemini CLI, and Cursor sessions in one platform
GitHub IntegrationOAuth SSO, PR sync, commit correlation, repository AI-readiness scoring

Quickstart

🎮 Want to see it first? demo.useprimer.dev is a live, read-only instance with sample data from a fictional 25-engineer org. No signup required.

One-liner install:

curl -fsSL https://useprimer.dev/install.sh | sh

Or install manually:

pip install useprimer            # Install (Python module is `primer`)
primer init                      # Initialize database and config
primer server start              # Start API + dashboard
primer hook install              # Register the SessionEnd hook

Docker:

cp .env.docker.example .env && make up

See the Installation guide for full setup, GitHub integration, and production configuration.

How It Works

AI agents + local capture ──POST──┐
GitHub App / webhooks ────────────┼──▶ Primer API ◀──MCP Sidecar
                                  │         │
                                  │         ▼
                                  └── PostgreSQL / SQLite
                                            │
                                            ▼
                                     React Dashboard
  1. Capture Layer — Hooks, sync, and import flows discover sessions across Claude Code, Codex CLI, Gemini CLI, and Cursor. Derived evidence includes facets, execution evidence, change shape, recovery paths, and workflow profiles.
  2. REST API — FastAPI service with routers covering analytics, alerting, FinOps, quality, maturity, interventions, explorer, and admin workflows. Role-based access is enforced across the platform.
  3. Dashboard — React frontend with organization, project, growth, quality, and individual views. Date-range filtering, CSV/PDF export, and role-based navigation are built in.
  4. MCP Sidecar — Engineers query their own data during sessions for stats, friction, recommendations, and coaching.

Documentation

GuideDescription
Getting StartedInstallationInstall, configure, first insights
ConfigurationEnvironment variables and options
CLI ReferenceAll primer commands
ArchitectureServer & APISystem design, data model, auth, endpoints
Hook SystemMulti-agent extractor registry
MCP SidecarMid-session stats, friction reports, recommendations
GuidesGitHub IntegrationOAuth login, GitHub App for PR sync
FinOps & Cost ManagementCache analytics, cost modeling, forecasting, budgets
Alert ThresholdsAnomaly detection and Slack notifications
DeploymentDocker Compose, Helm, PostgreSQL, scaling

About the Name

Named after the Young Lady's Illustrated Primer in Neal Stephenson's The Diamond Age — an adaptive, AI-driven book that observes its reader, understands context, and transforms complexity into personalized guidance. Primer brings that same principle to engineering organizations: observe how your team uses AI, understand the patterns, and guide each engineer toward mastery.

Contributing

We welcome contributions. See CONTRIBUTING.md for guidelines.

License

MIT

Ver o original no GitHub

Bom para

Relacionados