Follow the transformation
Evaluate predictions on data not used to fit/tune the model.
Choose metrics that match the target type and decision costs.
Inspect distributions/residuals or threshold curves rather than one number.
Evaluation & Diagnostics · Lesson 67
ROC and PR Curves
ROC and PR Curves is part of model evaluation.
What ROC and PR Curves actually means
ROC and PR Curves is part of model evaluation. A metric is a compressed view of model behaviour, so reliable evaluation uses several complementary summaries plus plots and subgroup/error analysis.
ROC and PR Curves matters because a single metric compresses model behaviour. Diagnostics, curves, residuals, calibration, thresholds and subgroup results reveal different failure modes that can lead to different deployment decisions.
Deeper walkthrough
Read ROC and PR Curves as a mechanism, not a recipe
Treat this as a sequence of observable decisions rather than one opaque command. Stage 1: Evaluate predictions on data not used to fit/tune the model. Stage 2: Choose metrics that match the target type and decision costs. Stage 3: Inspect distributions/residuals or threshold curves rather than one number. Final checkpoint: Attach uncertainty to performance estimates when sample size or variability matters.
Know what would convince you
- Recompute the metric from a tiny set of predictions/labels or residuals by hand.
- Inspect the underlying confusion counts, curve points, residuals or probability bins that generate the summary.
Useful distinctionAccuracy: Share of correct labels; can hide minority-class failure.
Click a stage to inspect what happens, what changes, and what should be checked before moving on.
Stage 1
Evaluate predictions on data not used…
Evaluate predictions on data not used to fit/tune the model. At this stage of ROC and PR Curves, keep the incoming data or object separate from the learned parameter, transformed object, or statistic so the change can be reproduced and independently checked.
Transformation focus: keep the input and produced parameters/result separate so the change is observable and reproducible.
Trace the mechanism step by step
- Evaluate predictions on data not used to fit/tune the model.
- Choose metrics that match the target type and decision costs.
- Inspect distributions/residuals or threshold curves rather than one number.
- Check subgroup performance and calibration when predictions drive decisions.
- Attach uncertainty to performance estimates when sample size or variability matters.
Make the concept concrete
Demonstration
Text example
At stricter thresholds: precision often rises while recall falls.
At looser thresholds: recall rises while precision may fall.
PR curves focus on positive predictions and are strongly affected by prevalence.Expected / illustrative result
Use the curve to select an operating region on validation data; evaluate the frozen threshold on held-out test data.
Interpret the result.
For ROC and PR Curves, connect the reported result to the exact training/validation/prediction step that produced it and check one prediction, fold or metric component independently.
Know what this is — and what it is not
AccuracyShare of correct labels; can hide minority-class failure.PrecisionAmong predicted positives, fraction truly positive.RecallAmong actual positives, fraction detected.ROC-AUCRanking across thresholds; may look optimistic under severe imbalance.PR-AUCPrecision-recall trade-off; often more informative for rare positives.MAE/RMSEAbsolute vs squared-error regression summaries.When it is appropriate
Use ROC and PR Curves when the metric/diagnostic corresponds to the prediction type, positive class or business/scientific consequence that matters.
When to stop or reconsider
Do not compress performance to this measure alone when thresholds, class imbalance, calibration, subgroup behaviour or error costs change the decision.
Failure modes to recognise
- Using a metric without defining the positive class, averaging rule, units or threshold that gives it meaning.
- Optimising/selecting on the final test set or changing the threshold after seeing test performance.
- Reporting one scalar while hiding calibration, residual structure, class imbalance or subgroup failures.
How to check the result
- Recompute the metric from a tiny set of predictions/labels or residuals by hand.
- Inspect the underlying confusion counts, curve points, residuals or probability bins that generate the summary.
- Evaluate the same frozen predictions across relevant thresholds/subgroups and explain which decision trade-off changes.
Demonstrate understanding
Try this:
Build a tiny, inspectable example of ROC and PR Curves. First evaluate predictions on data not used to fit/tune the model. Then choose metrics that match the target type and decision costs. Write the expected result before running it, and explain one condition that would make the result misleading or invalid.
Use 6–10 labelled predictions or residuals and calculate the measure manually. Keep the threshold/positive class/units explicit and connect the number to a decision.
Check reasoning, not memorisation
Before trusting a result from ROC and PR Curves, which check provides the strongest evidence that you understand and applied it correctly?
Keep the important distinctions visible
Step 1Evaluate predictions on data not used to fit/tune the model.Step 2Choose metrics that match the target type and decision costs.Step 3Inspect distributions/residuals or threshold curves rather than one number.Step 4Check subgroup performance and calibration when predictions drive decisions.What to remember
- ROC and PR Curves is part of model evaluation. A metric is a compressed view of model behaviour, so reliable evaluation uses several complementary summaries plus plots and subgroup/error analysis.
- Evaluate predictions on data not used to fit/tune the model.
- Using a metric without defining the positive class, averaging rule, units or threshold that gives it meaning.
- Recompute the metric from a tiny set of predictions/labels or residuals by hand.