convez.ai
SkillDados & AnalyticsAvançado
LeihuaYe

Análise Confiável de Testes A/B

LeihuaYe/claude-experimentation

Pipeline de gates para A/B tests: SRM → CUPED → efeito → BH → verdito

Ver no GitHub
InicialPluginScore · de 0 a 100

Skill Python para leitura rigorosa de experimentos A/B, implementando um pipeline de validação que inclui detecção de SRM (Sample Ratio Mismatch), redução de variância via CUPED, cálculo de efeitos com IC 95%, correção de múltiplos testes (Benjamini-Hochberg) e veredito final ship/no-ship. Evita falsos positivos e garante que resultados "significativos" sejam realmente confiáveis.

Instalação

$npx convez-ai LeihuaYe/claude-experimentation

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Validar se randomização de experimento foi bem-sucedida antes de ler resultados
  • Reduzir variância usando covariáveis pré-experimento (CUPED) sem custo
  • Controlar FDR (taxa de falsa descoberta) em painéis de múltiplas métricas
  • Tomar decisão ship/no-ship conservadora baseada em pipeline de checks, não p-value isolado

Veredito

Ferramenta production-grade para times de experimentos que precisam de rigor estatístico. Implementa boas práticas raramente vistas (SRM, CUPED, BH) em um workflow prático. Recomendado para quem quer evitar falsos positivos e ships arriscados.

README

A tradução para o português deste README ainda está na fila.

claude-experimentation

Trustworthy experiment analysis as Claude Code skills (and small, dependency-light Python libraries) — covering an experiment end to end:

  • ab-design — size it before it runs: sample size / MDE / power / duration, with CUPED, clustering, ratio metrics, and multiple-metric alpha.
  • ab-readout — read it out after it runs, as a pipeline of gates, not a single p-value.
  • ab-cate — go beyond the average: heterogeneous effects (S/T/X-learners), regression-adjusted ATE, and a subgroup-fishing guard so segment stories aren't false positives.

Most experiments are lost before they start (underpowered) or misread when they end (a significant result on a broken test). These skills check the experiment at both ends.

Why this exists

Most A/B readouts skip the steps that decide whether the number means anything:

StepQuestion it answersWhat goes wrong without it
1. SRMWas randomization even valid?A 50.4/49.6 split at scale silently invalidates the whole test
2. CUPEDCan we cut the noise for free?You ship a "flat" result that was actually a win hidden under variance
3. Effect + CIHow big, how sure?A bare p-value hides the effect size and the uncertainty
4. Benjamini-HochbergAre we fooling ourselves across metrics?The best of 8 metrics is "significant" by chance
5. VerdictShip or not?Cherry-picked metrics, ignored guardrails

Install

pip install -r requirements.txt   # numpy, scipy, pandas

Use it

CLI — point at a CSV:

python -m ab_readout \
  --data examples/example_experiment.csv \
  --arm group --primary revenue \
  --metrics revenue,engagement_min,conversion,latency_ms,support_tickets \
  --binary conversion \
  --guardrails latency_ms,support_tickets \
  --covariate pre_revenue

Library:

import pandas as pd
from ab_readout import run_readout, format_report

df = pd.read_csv("my_experiment.csv")
res = run_readout(df, arm_col="group", primary="revenue",
                  metrics=["revenue", "conversion"], binary=["conversion"],
                  guardrails=["latency_ms"], covariate="pre_revenue")
print(format_report(res))   # or use res (a dict) programmatically

As a Claude Code skill: drop SKILL.md into ~/.claude/skills/ab-readout/ and ask Claude to "read out this experiment" — it gathers the columns, runs the pipeline, and interprets the result conservatively (SRM failure halts the readout).

Example output

[1] SAMPLE RATIO MISMATCH  ->  PASS — randomization looks valid
[2] CUPED  corr=+0.599  variance reduction=35.9%  (CI ~20% narrower, same users)
[3] PRIMARY EFFECT
      raw    rel=+1.73%  95% CI [+0.0586, +0.2853]  p=0.0029
      CUPED  rel=+1.68%  95% CI [+0.0770, +0.2585]  p=0.0003
[4] METRIC PANEL + BH FDR
      revenue   [primary]  p=0.0003  <-- discovery
      engagement_min       p=0.0423  (does NOT survive BH)
      ...guardrails clean
[5] VERDICT  ->  SHIP

Full output in examples/example_output.txt.

Design the experiment first (ab-design)

A readout is only trustworthy if the test was powered. ab-design sizes it before launch — sample size, MDE, power, and duration are one quadrilateral; fix any three and it computes the fourth.

# how many users to detect a 0.1 lift on a mean metric, with a CUPED covariate?
python -m ab_design --kind mean --baseline 10 --sd 5 --mde 0.1 --rho 0.6 --daily 8000

# proportion metric, clustered randomization, 3 primary metrics
python -m ab_design --kind proportion --baseline 0.10 --mde 0.02 \
  --cluster-size 12 --icc 0.05 --k-primary 3
from ab_design import sample_size, mde, format_design_report

res = sample_size("mean", baseline=10, mde=0.1, sd=5, rho=0.6)   # CUPED cuts n by 1-rho^2
print(format_design_report(res, daily_users_per_arm=8000))

mde("proportion", baseline=0.10, n=20000)   # inverse: what can I detect with the traffic I have?

It handles the design realities that actually move required n: CUPED (covariate cuts n by 1−ρ²), clustered randomization (design effect 1+(m−1)·ICC), ratio metrics (delta-method variance), and multiple primary metrics (alpha allocation). Validated against Monte-Carlo power — python tests/test_design.py (12/12). Hand the result to ab-readout at the pre-registered n.

Go beyond the average effect (ab-cate)

The ATE hides the story — and slicing by segment until something turns green is how analysts fool themselves. ab-cate estimates how the effect varies and guards against fished subgroups.

python -m ab_cate --data exp.csv --arm group --outcome revenue \
  --covariates pre_revenue,tenure_days --learner x --subgroup-col country
from ab_cate import lin_estimator, x_learner, subgroup_fishing_guard, cate_summary

lin = lin_estimator(X, T, Y)                 # regression-adjusted ATE + variance reduction (CUPED's cousin)
summ = cate_summary(x_learner(X, T, Y), X)   # CATE spread: is there real heterogeneity?
guard = subgroup_fishing_guard(T, Y, {"power_users": mask_a, "new_users": mask_b})

It gives the regression-adjusted ATE (Lin's estimator — unbiased and lower-variance), S/T/X-learner CATE estimates, honest (sample-split) estimation, and a subgroup-fishing guard that Benjamini-Hochberg-corrects across every slice you tried — so "it worked better for X" has to survive the correction. Validated against known heterogeneity — python tests/test_cate.py (9/9).

The stats are checked, not just plausible

The methods are validated against a simulation with a known ground truth — see tests/test_readout.py: CUPED variance reduction lands near ρ², the primary CI covers the true effect, BH demotes the true nulls, SRM catches a real imbalance.

python tests/test_readout.py    # 8/8 passed

Methods

  • SRM — chi-square goodness-of-fit on the assignment split (α=0.001, because it's a data-quality alarm).
  • CUPED — Y_adj = Y − θ(X − E[X]), θ = Cov(Y,X)/Var(X); variance falls ~ρ², CIs shrink for free.
  • Effect — Welch (unequal-variance) for means, two-proportion z for binary, with 95% CIs.
  • Benjamini-Hochberg — FDR control across the metric panel so the best borderline metric isn't a false positive.

Roadmap

ab-design, ab-readout, and ab-cate ship today. Planned: sequential / always-valid inference (peeking-safe stopping) and causal inference without randomization (DiD / synthetic control).

License

MIT — see LICENSE. Built by Leihua Ye.

Ver o original no GitHub

Bom para

Relacionados