Core Principles
| Item | Description |
|---|
Randomization | Assign subjects to conditions randomly — eliminates systematic bias |
Replication | Multiple subjects per condition — estimates experimental error, increases power |
Blocking | Group similar subjects together — accounts for known nuisance variation |
Control | Baseline group receiving no treatment or current standard — essential for comparison |
Blinding | Single-blind (subject doesn't know group) or double-blind (neither subject nor experimenter) |
Balance | Equal (or proportional) sample sizes across groups — maximizes power |
Experimental Designs
| Design | Description | Pros | Cons |
|---|
| Completely Randomized | Subjects randomly assigned to treatments | Simple, valid | Low precision with high variability |
| Randomized Block | Subjects blocked by covariate, then randomized within each block | Controls known variation | Must know blocking variable |
| Factorial (2^k) | Test k factors each at 2 levels — all combinations | Tests interactions efficiently | Grows exponentially with factors |
| Fractional Factorial | Subset of full factorial — 2^(k-p) runs | Reduces runs for many factors | Confounds higher-order interactions |
| Latin Square | Two blocking factors (row, column) | Controls 2 nuisance variables | Assumes no interactions |
| Crossover | Each subject receives multiple treatments sequentially | Each subject is own control | Carryover effects possible |
| Split-Plot | Hard-to-change factors at whole-plot level; easy-to-change at subplot | Practical for industrial settings | Complex analysis |
Power Analysis
| Item | Description |
|---|
Statistical Power | Probability of detecting an effect IF it exists — aim for 80% minimum |
Power depends on | Effect size, sample size, significance level (alpha), and variability |
Cohen's d (Effect Size) | Standardized mean difference: (mu1 - mu2) / sigma |
Effect Sizes: 0.2 = small, 0.5 = medium, 0.8 = large | Benchmarks — context matters; domain-specific standards vary |
G*Power / statsmodels | Software for power analysis — calculate required n before experiment |
Post-Hoc Power | Generally discouraged — trust the confidence interval width instead |
Sample Size Determination
| Item | Description |
|---|
Required n per group | n = 2*(z_alpha + z_beta)^2 * sigma^2 / delta^2 for two-group comparison |
Continuous outcome, two groups | Larger sigma = more subjects; larger delta to detect = fewer subjects |
Attrition Adjustment | Plan for 10-20% dropout — inflate sample size accordingly |
Pilot Study for Estimates | Use pilot data to estimate sigma and refine sample size calculations |
Equivalence/Non-Inferiority | Different formulas — proving no difference requires larger samples |
Pro Tip: The three pillars of good experiments: Randomization (eliminates bias), Replication (estimates error), and Blocking (reduces variability). Without all three, your results are questionable.