Linear Regression
| Item | Description |
|---|
Model: y = beta0 + beta1*x + epsilon | Simple linear regression — one predictor, intercept, error term |
OLS Estimate: beta1 = Cov(x,y)/Var(x) | Ordinary Least Squares — minimizes sum of squared residuals |
R-squared = 1 - SS_residual / SS_total | Proportion of variance explained: 0 to 1, higher is better fit |
Adjusted R-squared | Penalizes adding useless predictors: Adj R^2 = 1 - ((1-R^2)(n-1)/(n-k-1)) |
RMSE = sqrt(Mean((y - yhat)^2)) | Root Mean Squared Error — interpretable in original units |
Multiple Regression: y = X*beta + epsilon | Multiple predictors — matrix form, same OLS principle |
OLS Assumptions (LINE)
| Item | Description |
|---|
L - Linearity | Relationship between X and y is linear in parameters |
I - Independence | Observations are independent (no clustering, no autocorrelation) |
N - Normality | Residuals are normally distributed (important for inference, less for prediction) |
E - Equal Variance (Homoscedasticity) | Constant variance of residuals across fitted values |
Multicollinearity Warning | Predictors highly correlated — inflates standard errors; check VIF > 10 |
Outliers & Leverage | Cook's distance > 1 flags influential points — investigate before removing |
Logistic Regression
| Item | Description |
|---|
Model: P(y=1) = 1 / (1 + e^(-z)) where z = beta0 + beta1*x | Binary classification — output is probability between 0 and 1 |
Log-Odds: ln(P/(1-P)) = z | Linear relationship in log-odds space — the link function |
Coefficient = Log Odds Ratio | e^beta = odds multiplier per unit increase in predictor |
Maximum Likelihood (not OLS) | Iterative fitting — no closed-form solution |
Metrics: Accuracy, Precision, Recall, AUC | Don't use R-squared; threshold matters for classification |
Multiclass Extension | One-vs-rest or softmax (multinomial logistic regression) |
Polynomial & Advanced
| Item | Description |
|---|
Polynomial: y = beta0 + beta1*x + beta2*x^2 + ... | Captures curves — degree d polynomial fits d bends |
Overfitting Warning | Higher degree = more wiggly; cross-validate to choose degree |
Regularized: Ridge (L2), Lasso (L1) | Penalty on coefficient size — Lasso does feature selection by zeroing betas |
Interaction Terms: y = ... + beta_ij*x_i*x_j | Effect of one predictor depends on another — synergistic effects |
Pro Tip: Always plot residuals vs fitted values before trusting any regression. Patterns (funnel shapes, curves) mean your model is misspecified — no amount of R-squared can save it.