Entropy
| Item | Description |
|---|
Shannon Entropy: H(X) = -Sum(p(x)*log(p(x))) | Measure of uncertainty/surprise — average bits needed to encode |
H(X) >= 0 | Entropy is always non-negative; 0 for deterministic (one outcome p=1) |
H_max = log(n) | Maximum entropy when all n outcomes equally likely — uniform distribution |
Binary Entropy: H(p) = -p*log(p) - (1-p)*log(1-p) | Entropy of Bernoulli(p) — peaks at p=0.5 |
log base 2 | Bits; base e = nats; base 10 = hartleys — base 2 is standard in ML |
High Entropy | Lot of uncertainty — flat distribution; Low entropy = confident, peaked |
KL Divergence
| Item | Description |
|---|
D_KL(P||Q) = Sum(P(x)*log(P(x)/Q(x))) | How much information is lost when Q approximates P |
D_KL >= 0 | Gibbs inequality — always non-negative; zero only when P=Q |
NOT symmetric: D_KL(P||Q) != D_KL(Q||P) | Direction matters — NOT a distance metric |
Forward KL: D_KL(P||Q) | Mean-seeking — Q spreads to cover all of P (avoiding zeros) |
Reverse KL: D_KL(Q||P) | Mode-seeking — Q concentrates on high density regions of P |
In Variational Inference | Minimize KL(q_theta || p_posterior) to find approximate posterior |
Mutual Information & Cross-Entropy
| Item | Description |
|---|
I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X) | How much knowing X reduces uncertainty about Y — symmetric |
I(X;Y) = D_KL(P(X,Y) || P(X)P(Y)) | Mutual information = KL divergence from independence |
Feature Selection with MI | Select features with highest mutual information with target |
Cross-Entropy: H(P,Q) = -Sum(P(x)*log(Q(x))) | Average bits to encode P using model Q — classification loss |
H(P,Q) = H(P) + D_KL(P||Q) | Cross-entropy = entropy of truth + divergence from truth |
Softmax + Cross-Entropy | Standard combination in neural nets for multi-class classification |
ML Applications
| Item | Description |
|---|
Entropy in Decision Trees | Information gain = H(parent) - weighted H(children) — split criterion |
Cross-Entropy Loss | Dominant loss for classification — differentiable, well-behaved gradients |
VAE: KL divergence in latent space | VAE loss = reconstruction + D_KL(q(z|x) || p(z)) |
Perplexity = 2^H | Exponential of cross-entropy — measures model uncertainty in language models |
Pro Tip: Cross-entropy loss is the standard loss function for classification because minimizing it is equivalent to maximizing the likelihood of the true class labels under your model.