Gradient Descent
| Item | Description |
|---|
Batch GD: w = w - lr * grad_w(Loss(all_data)) | Computes gradient on entire dataset — slow but stable |
Stochastic GD (SGD): w = w - lr * grad_w(Loss(single_point)) | One sample at a time — very noisy, faster updates |
Mini-Batch SGD | Compromise: gradient on small batch (32-256 samples) — standard in deep learning |
Momentum: v_t = beta*v_(t-1) + lr*grad_t | Accumulates velocity — smooths oscillations, accelerates in consistent directions |
Nesterov Accelerated Gradient | Look ahead: compute gradient at future position for smarter moves |
Learning Rate Schedules | Step decay, exponential decay, cosine annealing — reduce lr over time |
Adam Optimizer
| Item | Description |
|---|
Adam = Momentum + RMSProp | Adaptive learning rate for each parameter — most popular optimizer |
m_t = beta1*m_(t-1) + (1-beta1)*grad_t | First moment estimate (mean of gradients) — momentum term |
v_t = beta2*v_(t-1) + (1-beta2)*grad_t^2 | Second moment estimate (uncentered variance) — scales learning rate |
Default: beta1=0.9, beta2=0.999, lr=0.001 | These hyperparameters work well for most problems |
Bias Correction | Corrects for initial zero bias in moment estimates during early steps |
AdamW (Weight Decay) | Decoupled weight decay — better generalization than standard Adam |
Convex vs Non-Convex
| Item | Description |
|---|
Convex Function | Any local minimum is global — bowl-shaped, gradient descent guaranteed to converge |
Convex Examples | Linear regression, logistic regression, SVM — nice optimization landscapes |
Non-Convex Examples | Neural networks, mixture models — many local minima, saddle points |
Saddle Points | Gradient = 0 but not a minimum — Adam's momentum helps escape |
Local Minima | Deep nets: most local minima are close to global in quality (surprising empirical result) |
Checking Convergence | Plateau in loss, gradient norm ~ 0, validation metrics stop improving |
Practical Tips
| Item | Description |
|---|
Learning Rate is Most Important | Tune lr first; bad lr prevents convergence regardless of other hyperparameters |
Gradient Clipping | Limit gradient magnitude — prevents exploding gradients in RNNs |
Batch Normalization | Smooths loss landscape, allows higher learning rates |
Early Stopping | Stop training when validation loss stops improving — prevents overfitting |
Loss Landscape Visualization | Plot loss as a function of parameters in 2D slices to understand geometry |
Pro Tip: Start with a learning rate of 0.001 and adjust by factors of 3-10. If loss oscillates or diverges, reduce it. If loss decreases too slowly, increase it. Monitor the LOSS CURVE, not just final numbers.