Predictive Uncertainty
Uncertainty estimation distinguishes what the model predicts from how confident it should be, including noise intrinsic to data and uncertainty due to limited model knowledge. The topic is split into focused lessons so definitions, implementation decisions, diagnostics and common failure modes can be learned separately.
Learn the mechanism one decision at a time
Work through the lessons in order if the topic is new. If you already know the basics, open the specific leaf lesson that matches the operation, diagnostic or failure mode you need.
1Definition→
2Mechanism→
3Example→
4Diagnostic→
5Decision
Aleatoric uncertaintyIrreducible variability or measurement noise in the data-generating process. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.
02Epistemic uncertaintyUncertainty about model parameters/functions due to limited evidence; it may reduce with informative new data. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.
03Predictive intervalsRegression intervals communicate a range of plausible outcomes; coverage should be validated empirically. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.
04Bayesian and ensemble approachesGaussian Processes, Bayesian neural methods, deep ensembles and MC dropout approximate distributions over predictions. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.
05Conformal predictionUses calibration residuals/scores to create sets or intervals with finite-sample coverage guarantees under exchangeability assumptions. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.