Covariance
| Item | Description |
|---|
Cov(X,Y) = E[(X - mu_X)(Y - mu_Y)] | How X and Y vary together — sign indicates direction of relationship |
Cov > 0 | X and Y tend to increase together — positive relationship |
Cov < 0 | X above mean when Y below mean — negative relationship |
Limitation | Scale-dependent — changing units changes covariance value |
Cov(X,X) = Var(X) | Covariance of variable with itself is its variance |
Covariance Matrix | Symmetric matrix — diagonal = variances, off-diagonal = pairwise covariances |
Pearson Correlation
| Item | Description |
|---|
r = Cov(X,Y) / (sigma_X * sigma_Y) | Normalized covariance — dimensionless, ranges from -1 to +1 |
r = 1.0 | Perfect positive linear relationship |
r = -1.0 | Perfect negative linear relationship |
r = 0 | No linear relationship (but could have nonlinear relationship) |
Strength Guidelines | |r|<0.3=weak, 0.3-0.5=moderate, 0.5-0.7=strong, >0.7=very strong |
Assumptions | Linear relationship, continuous variables, no outliers, homoscedasticity |
Spearman Rank Correlation
| Item | Description |
|---|
rs = Pearson on ranked data | Measures monotonic (non-necessarily linear) relationship |
Advantage over Pearson | Robust to outliers and non-linear monotonic relationships |
Use When | Ordinal data, outliers present, monotonic but not linear relationship |
Interpretation | Same -1 to +1 range as Pearson, but about rank agreement |
Tied Ranks | Spearman handles ties with average rank assignment |
Correlation with ordinal | Spearman is preferred for Likert scales and ordinal categories |
Correlation Pitfalls
| Item | Description |
|---|
Spurious Correlation | Two unrelated variables correlated by chance or common cause |
Anscombe's Quartet | Four datasets with identical correlations but vastly different patterns |
Simpson's Paradox | Trend appears in groups but reverses when groups combined |
Restriction of Range | Narrow range of values artificially reduces observed correlation |
Correlation Matrix Visualization | Heatmap makes patterns obvious — use it before modeling |
Pro Tip: Correlation does not imply causation. Confounding variables, reverse causation, and coincidence all produce correlations. Always consider the causal mechanism.