Vectors
| Item | Description |
|---|
Vector: v = [v1, v2, ..., vn] | Ordered list of numbers — point in n-dimensional space |
Dot Product: v*w = Sum(v_i * w_i) | Scalar — measures alignment: positive=same direction, 0=orthogonal |
Norm (L2): ||v||_2 = sqrt(Sum(v_i^2)) | Euclidean length of vector — most common distance |
Norm (L1): ||v||_1 = Sum(|v_i|) | Manhattan distance — sum of absolute values, sparse-promoting |
Cosine Similarity = (v*w) / (||v||*||w||) | Similarity measure: 1.0 = same direction, 0 = orthogonal, -1 = opposite |
Unit Vector = v / ||v|| | Vector with length 1 — preserves direction, normalizes scale |
Matrices
| Item | Description |
|---|
Matrix: A with m rows, n columns | m x n grid of numbers — linear transformation |
Matrix Addition | Element-wise — requires same dimensions |
Matrix Multiplication: C = A*B | Rows of A dot columns of B — A(mxk) * B(kxn) = C(mxn) |
Transpose: A^T | Rows become columns: (A^T)_ij = A_ji |
Inverse: A^(-1) | Matrix such that A*A^(-1) = I — only for square, full-rank matrices |
Determinant: det(A) | Scalar — zero means singular (non-invertible); measures area/volume scaling |
Eigenvalues & SVD
| Item | Description |
|---|
Eigenvalue Equation: Av = lambda*v | A transforms v by only scaling it (no rotation) |
Eigendecomposition: A = V*Lambda*V^(-1) | For diagonalizable matrices — A expressed in eigenbasis |
Singular Value Decomposition: A = U*Sigma*V^T | Works for ANY matrix — U/V=orthogonal, Sigma=diagonal singular values |
SVD Interpretation | U: left singular vectors (output basis), Sigma: importance weights, V^T: right singular vectors (input basis) |
Truncated SVD / Low-Rank Approx | Keep top-k singular values — best rank-k approximation (Eckart-Young) |
PCA = SVD on centered data | Principal components = right singular vectors of centered data matrix |
Common Operations in ML
| Item | Description |
|---|
C = A @ B (Python: @ or dot) | Matrix multiplication — core of neural network forward passes |
Orthogonal Matrix: A^T*A = I | Columns are orthonormal — preserves lengths (SVD, QR) |
Trace: tr(A) = Sum(A_ii) | Sum of diagonal elements — invariant under cyclic permutations |
Rank | Number of linearly independent rows/columns — dimensionality of column space |
Pro Tip: SVD is the swiss army knife of linear algebra — it underlies PCA, pseudoinverses, low-rank approximations, and recommendation systems.