Test Selection by Data Type
| Your Data | Goal | Parametric Test | Non-Parametric Alternative |
|---|
| 1 group continuous vs known value | Compare to population | One-sample t-test | Wilcoxon signed-rank |
| 2 independent groups continuous | Compare means | Independent t-test | Mann-Whitney U |
| 2 paired groups continuous | Compare paired | Paired t-test | Wilcoxon signed-rank |
| 3+ independent groups continuous | Compare means | One-way ANOVA | Kruskal-Wallis |
| Repeated measures (3+ times) | Compare over time | Repeated ANOVA | Friedman test |
| 2 categorical variables | Association | Chi-square | Fisher's exact (small n) |
| Correlation (continuous) | Strength/association | Pearson r | Spearman rho / Kendall tau |
| 1 categorical (2 levels) + continuous | Predict group | Logistic regression | — |
Parametric vs Non-Parametric
| Item | Description |
|---|
Parametric Assumptions | Normality, homogeneity of variance, continuous interval/ratio data |
Parametric Advantage | More statistical power if assumptions met — detect smaller effects |
Non-Parametric Advantage | No distribution assumptions — valid for ordinal, skewed, small samples |
When to Choose Non-Parametric | Ordinal data, outliers present, small n, visibly non-normal distributions |
Median vs Mean | Non-parametric tests often compare medians, not means |
Mann-Whitney U | Tests if one group tends to have larger values — not a test of medians per se |
Normality Checks
| Item | Description |
|---|
Shapiro-Wilk Test | Formal test for normality — H0: data is normal; significant = non-normal |
Q-Q Plot | Graphical check — points should follow diagonal line |
Kolmogorov-Smirnov | Tests if sample comes from a specific distribution |
Central Limit Theorem | With n>30, sampling distribution of mean ~ normal even if data isn't |
Anderson-Darling | Modified normality test — more sensitive in tails than Shapiro-Wilk |
Common Pitfalls
| Item | Description |
|---|
Multiple Testing | Running many tests inflates Type I error — use Bonferroni or FDR correction |
P-Hacking | Trying different tests/rejecting until p<0.05 invalidates the p-value |
Practical vs Statistical Significance | Very small effects can be 'significant' with huge n |
Choosing Test After Seeing Data | The test must be chosen BEFORE data analysis — otherwise, inflated error |
Pro Tip: The most common mistake: using a parametric test on non-normal data. If data is skewed or ordinal, use non-parametric tests — they sacrifice some power for validity.