Follow the transformation
Evaluate predictions on data not used to fit/tune the model.
Choose metrics that match the target type and decision costs.
Inspect distributions/residuals or threshold curves rather than one number.
Evaluation & Diagnostics · Lesson 72
Uncertainty
Uncertainty is part of model evaluation.
What Uncertainty actually means
Uncertainty is part of model evaluation. A metric is a compressed view of model behaviour, so reliable evaluation uses several complementary summaries plus plots and subgroup/error analysis.
Uncertainty matters because a single metric compresses model behaviour. Diagnostics, curves, residuals, calibration, thresholds and subgroup results reveal different failure modes that can lead to different deployment decisions.
Deeper walkthrough
Read Uncertainty as a mechanism, not a recipe
Treat this as a sequence of observable decisions rather than one opaque command. Stage 1: Evaluate predictions on data not used to fit/tune the model. Stage 2: Choose metrics that match the target type and decision costs. Stage 3: Inspect distributions/residuals or threshold curves rather than one number. Final checkpoint: Attach uncertainty to performance estimates when sample size or variability matters.
Know what would convince you
- Recompute the metric from a tiny set of predictions/labels or residuals by hand.
- Inspect the underlying confusion counts, curve points, residuals or probability bins that generate the summary.
Useful distinctionAccuracy: Share of correct labels; can hide minority-class failure.
Trace the mechanism step by step
- Evaluate predictions on data not used to fit/tune the model.
- Choose metrics that match the target type and decision costs.
- Inspect distributions/residuals or threshold curves rather than one number.
- Check subgroup performance and calibration when predictions drive decisions.
- Attach uncertainty to performance estimates when sample size or variability matters.
Make the concept concrete
Demonstration
Text example
Point estimate: expected demand = 1,000 units.
Uncertainty statement: plausible forecast range is 850–1,180 under current conditions.
Limitation: the range does not cover a structural break such as a new regulation or supply shock.Expected / illustrative result
A useful report separates estimated value, quantified uncertainty and unmodelled limitations.
Interpret the result.
For Uncertainty, connect the reported result to the exact training/validation/prediction step that produced it and check one prediction, fold or metric component independently.
Know what this is — and what it is not
AccuracyShare of correct labels; can hide minority-class failure.PrecisionAmong predicted positives, fraction truly positive.RecallAmong actual positives, fraction detected.ROC-AUCRanking across thresholds; may look optimistic under severe imbalance.PR-AUCPrecision-recall trade-off; often more informative for rare positives.MAE/RMSEAbsolute vs squared-error regression summaries.When it is appropriate
Use Uncertainty when the metric/diagnostic corresponds to the prediction type, positive class or business/scientific consequence that matters.
When to stop or reconsider
Do not compress performance to this measure alone when thresholds, class imbalance, calibration, subgroup behaviour or error costs change the decision.
Failure modes to recognise
- Using a metric without defining the positive class, averaging rule, units or threshold that gives it meaning.
- Optimising/selecting on the final test set or changing the threshold after seeing test performance.
- Reporting one scalar while hiding calibration, residual structure, class imbalance or subgroup failures.
How to check the result
- Recompute the metric from a tiny set of predictions/labels or residuals by hand.
- Inspect the underlying confusion counts, curve points, residuals or probability bins that generate the summary.
- Evaluate the same frozen predictions across relevant thresholds/subgroups and explain which decision trade-off changes.
Demonstrate understanding
Try this:
Build a tiny, inspectable example of Uncertainty. First evaluate predictions on data not used to fit/tune the model. Then choose metrics that match the target type and decision costs. Write the expected result before running it, and explain one condition that would make the result misleading or invalid.
Use 6–10 labelled predictions or residuals and calculate the measure manually. Keep the threshold/positive class/units explicit and connect the number to a decision.
Check reasoning, not memorisation
Before trusting a result from Uncertainty, which check provides the strongest evidence that you understand and applied it correctly?
Keep the important distinctions visible
Step 1Evaluate predictions on data not used to fit/tune the model.Step 2Choose metrics that match the target type and decision costs.Step 3Inspect distributions/residuals or threshold curves rather than one number.Step 4Check subgroup performance and calibration when predictions drive decisions.What to remember
- Uncertainty is part of model evaluation. A metric is a compressed view of model behaviour, so reliable evaluation uses several complementary summaries plus plots and subgroup/error analysis.
- Evaluate predictions on data not used to fit/tune the model.
- Using a metric without defining the positive class, averaging rule, units or threshold that gives it meaning.
- Recompute the metric from a tiny set of predictions/labels or residuals by hand.