Bayes Theorem
| Item | Description |
|---|
P(A|B) = P(B|A) * P(A) / P(B) | The fundamental formula — update belief A given evidence B |
Posterior = (Likelihood * Prior) / Evidence | Interpret: What you believe after data = what data says * what you believed before |
Prior P(H) | Your belief about hypothesis H before seeing data |
Likelihood P(D|H) | Probability of observing data D if hypothesis H is true |
Posterior P(H|D) | Updated belief about H after observing data D |
Marginal P(D) = Integral P(D|H)*P(H) dH | Normalizing constant — often intractable, hence MCMC |
Priors & Posteriors
| Item | Description |
|---|
Conjugate Priors | Prior and posterior are same distribution family — closed-form updates |
Beta-Binomial: Beta(a,b) prior | Conjugate for binomial likelihood; posterior = Beta(a+successes, b+failures) |
Normal-Normal: N(mu0, tau0^2) prior | Conjugate for normal with known variance; convenient updating |
Uninformative/Flat Prior | No strong prior belief — let data dominate: P(H) ~ constant |
Informative Prior | Encodes expert knowledge or previous studies — controversial in some fields |
Jeffreys Prior | Objective prior that is invariant under reparameterization |
MCMC Sampling
| Item | Description |
|---|
Why MCMC? | Posterior rarely has closed form — MCMC draws samples to approximate it |
Metropolis-Hastings | Propose new parameter value, accept/reject based on posterior ratio |
Gibbs Sampling | Sample each parameter conditional on others — efficient for conjugates |
Hamiltonian Monte Carlo (HMC) | Uses gradient info — Stan/PyMC default, much faster for high dimensions |
NUTS (No U-Turn Sampler) | Auto-tunes HMC step size — the standard in modern Bayesian software |
Convergence Diagnostics | R-hat close to 1, trace plots stationary, effective sample size sufficient |
Credible Intervals vs Confidence Intervals
| Item | Description |
|---|
95% Credible Interval | 95% probability the parameter is in this interval (given data + prior) |
95% Confidence Interval | If we repeated the study 100 times, 95 intervals would contain true value |
HDI (Highest Density Interval) | Narrowest interval containing 95% of posterior mass |
Bayesian Hypothesis Testing | Compare posterior probabilities of competing models directly |
Pro Tip: The biggest Bayesian insight: parameters are random variables, not fixed constants. You express uncertainty about them with probability distributions — not just point estimates.