PCA (Principal Component Analysis)
| Item | Description |
|---|
Goal | Find orthogonal directions (principal components) that maximize variance |
Math: Eigenvalues of Covariance Matrix | Top-k eigenvectors = k principal components |
Or: SVD of centered data matrix | Singular vectors = principal components; singular values = sqrt(variance) |
Explained Variance Ratio | lambda_i / Sum(lambda_j) — proportion of variance captured by component i |
Cumulative Explained Variance | Sum of top-k variance ratios — how much information is retained |
Choosing k | Elbow method, 95% variance threshold, or domain requirements |
Standardize First! | PCA is scale-sensitive — standardize to mean 0, variance 1 before PCA |
t-SNE
| Item | Description |
|---|
Goal | Visualize high-dimensional data in 2D/3D — preserves local neighborhoods |
How It Works | Converts high-D distances to probabilities; minimizes KL divergence in low-D |
Perplexity | Balances local vs global structure — typical range 5-50 |
Low Perplexity (5-15) | Focus on very local structure — fine details, small clusters |
High Perplexity (30-50) | More global view — larger-scale patterns |
Iterations | Needs 1,000+ — watch for convergence; early exaggeration for initial separation |
Random Seed Matters | Different seeds give different results — t-SNE is non-deterministic |
Limitations | Cannot interpret axis meanings; distances between clusters are meaningless |
UMAP
| Item | Description |
|---|
Goal | Faster, more scalable t-SNE alternative — preserves global and local structure |
Key Parameter: n_neighbors | Controls local vs global balance — similar to perplexity |
min_distance | How tightly points are packed — low=clumpy, high=spread out |
Speed Advantage | Much faster than t-SNE for large datasets (thousands to millions of points) |
Deterministic (with fixed seed) | Unlike t-SNE — reproducible results |
General Purpose | Can be used for feature reduction (not just visualization, unlike t-SNE) |
When to Use Each
| Method | Best For | Speed | Interpretability | Scale |
|---|
| PCA | Linear patterns, preprocessing, feature extraction | Fast | High (components=linear combos) | Any |
| t-SNE | Visualizing clusters, local structure | Slow | Low (axes meaningless) | <10K samples typical |
| UMAP | Visualization + general reduction | Fast | Medium | Scales to millions |
Pro Tip: PCA excels at preserving global structure for linear relationships. t-SNE preserves local neighborhoods for visualization. UMAP balances both. Use PCA for preprocessing; t-SNE/UMAP for visualization.