Sample Size Calculation
| Item | Description |
|---|
n = (z_alpha + z_beta)^2 * p(1-p) / delta^2 | Sample size per variant — binary metric (conversion rate) |
z_alpha (Type I) | z for significance level: 1.96 for 95% confidence (two-sided) |
z_beta (Type II) | z for desired power: 0.84 for 80% power, 1.28 for 90% power |
Baseline Rate (p) | Current conversion rate — higher variance near 50% |
Minimum Detectable Effect (delta) | Smallest improvement worth detecting — set by business requirements |
For continuous metrics | n = 2*(z_alpha + z_beta)^2 * sigma^2 / delta^2 |
Statistical Power
| Item | Description |
|---|
Power = P(reject H0 | H1 is true) | Probability of detecting a real effect — 1 - Type II error |
Typical Power Target: 80% | Industry standard — 20% chance of missing a real effect |
90% Power | More conservative — requires ~30% larger sample size |
Power Analysis | Run BEFORE test: determine required sample size for desired power |
Factors Affecting Power | Effect size, sample size, significance level, variance |
Underpowered Studies | Waste resources — likely to miss real effects; ethical concerns |
Sequential Testing & Peeking
| Item | Description |
|---|
The Peeking Problem | Repeatedly checking p-values and stopping when significant inflates Type I error |
Sequential Testing Methods | Group sequential designs, alpha spending functions (O'Brien-Fleming) |
Always Valid p-values | Designed for continuous monitoring — martingale-based stopping bounds |
Fixed Horizon Rule | Pre-specify sample size and only evaluate at the end — simplest, safest |
Suggested Approach | Use a sequential testing framework from the start OR commit to fixed horizon |
Practical Guidelines
| Item | Description |
|---|
Randomization Unit | User, session, device — ensure independence and avoid contamination |
Run Duration | At least one full business cycle (typically 1-2 weeks) to capture day-of-week effects |
Guardrail Metrics | Monitor key secondary metrics to detect negative side effects |
Multiple Variants | Bonferroni correction: alpha/m where m = number of comparisons |
Separate Analysis from Setup | The person running the test shouldn't decide when it ends |
Pro Tip: Never peek at results and stop early — the peeking problem inflates Type I error dramatically. Use sequential testing or fixed horizon. A test with 10 peeks at alpha=0.05 has ~20% false positive rate.