HumachLearnLearn · explore · apply
Start learning
Metrics, Calibration & Thresholds · Lesson 74

ROC AUC

ROC AUC evaluates a particular aspect of prediction quality.

ConceptWorked examplePracticeKnowledge check
Textbook walkthrough

What ROC AUC actually means

ROC AUC evaluates a particular aspect of prediction quality. No single metric captures discrimination, calibration, threshold costs and subgroup performance simultaneously; choose metrics based on the decision and class/base-rate structure.

ROC AUC matters because classification and regression metrics answer different operational questions. Ranking, probability quality, threshold decisions and error magnitudes should be evaluated separately when the application cares about them separately.

Deeper walkthrough

Read ROC AUC as a mechanism, not a recipe

Treat this as a sequence of observable decisions rather than one opaque command. Stage 1: Generate predictions on held-out data. Stage 2: For probabilistic classifiers, separate probability quality from thresholded decisions. Stage 3: Compute metrics with the positive class and averaging convention explicitly defined. Final checkpoint: Choose an operating threshold using validation data and decision costs, then freeze it for test evaluation.

Mechanism

Follow the transformation

Generate predictions on held-out data.

For probabilistic classifiers, separate probability quality from thresholded decisions.

Compute metrics with the positive class and averaging convention explicitly defined.

Evidence

Know what would convince you

  • Recompute the metric from a tiny set of predictions/labels or residuals by hand.
  • Inspect the underlying confusion counts, curve points, residuals or probability bins that generate the summary.
Useful distinctionPrecision: TP/(TP+FP): reliability of positive predictions.
Visual demonstration of ROC AUC
Visual demonstration: use the diagram to trace the main objects and state changes involved in ROC AUC.
Click a stage to inspect what happens, what changes, and what should be checked before moving on.
Stage 1

Generate predictions on held-out data

Generate predictions on held-out data. Treat the output from ROC AUC as evidence to inspect: confirm its type, shape, range or units and connect it back to the input that produced it.

Output focus: inspect both the value and its shape/type/meaning before treating it as a trustworthy result.
How it works

Trace the mechanism step by step

  1. Generate predictions on held-out data.
  2. For probabilistic classifiers, separate probability quality from thresholded decisions.
  3. Compute metrics with the positive class and averaging convention explicitly defined.
  4. Inspect a curve/confusion matrix/residual distribution in addition to scalar metrics.
  5. Choose an operating threshold using validation data and decision costs, then freeze it for test evaluation.
Worked demonstration

Make the concept concrete

Demonstration

Text example

Threshold ↓ : more cases predicted positive → both true-positive rate and false-positive rate generally rise.
ROC-AUC summarises ranking across all thresholds; it does not choose the deployment threshold for you.
Expected / illustrative result
A high ROC-AUC can coexist with poor calibration or unacceptable precision at the chosen operating point.
Interpret the result.

For ROC AUC, connect the reported result to the exact training/validation/prediction step that produced it and check one prediction, fold or metric component independently.

Distinctions & related ideas

Know what this is — and what it is not

PrecisionTP/(TP+FP): reliability of positive predictions.
RecallTP/(TP+FN): detection of actual positives.
F1/FβHarmonic combination of precision/recall; β weights recall relative to precision.
ROC-AUCProbability a random positive ranks above a random negative.
PR-AUCPrecision-recall ranking summary, sensitive to prevalence.
Brier/log lossProbability-quality scores that penalise miscalibrated/confident errors.
Use deliberately

When it is appropriate

Use ROC AUC when the metric/diagnostic corresponds to the prediction type, positive class or business/scientific consequence that matters.

Boundary conditions

When to stop or reconsider

Do not compress performance to this measure alone when thresholds, class imbalance, calibration, subgroup behaviour or error costs change the decision.

Common mistakes

Failure modes to recognise

  • Using a metric without defining the positive class, averaging rule, units or threshold that gives it meaning.
  • Optimising/selecting on the final test set or changing the threshold after seeing test performance.
  • Reporting one scalar while hiding calibration, residual structure, class imbalance or subgroup failures.
Verification

How to check the result

  • Recompute the metric from a tiny set of predictions/labels or residuals by hand.
  • Inspect the underlying confusion counts, curve points, residuals or probability bins that generate the summary.
  • Evaluate the same frozen predictions across relevant thresholds/subgroups and explain which decision trade-off changes.
Connection to feature importance

AUC as univariate feature evidence

ROC-AUC can also score one numeric feature at a time. Direction-adjusted AUC, max(AUC, 1 − AUC), measures how well that feature alone ranks the two classes; it is not the same as multivariate model importance.

Open Explainable AI Playground →Feature-selection workflow
Hands-on practice

Demonstrate understanding

Try this:

Build a tiny, inspectable example of ROC AUC. First generate predictions on held-out data. Then for probabilistic classifiers, separate probability quality from thresholded decisions. Write the expected result before running it, and explain one condition that would make the result misleading or invalid.

Use 6–10 labelled predictions or residuals and calculate the measure manually. Keep the threshold/positive class/units explicit and connect the number to a decision.
Knowledge check

Check reasoning, not memorisation

Before trusting a result from ROC AUC, which check provides the strongest evidence that you understand and applied it correctly?

Quick reference

Keep the important distinctions visible

Step 1Generate predictions on held-out data.
Step 2For probabilistic classifiers, separate probability quality from thresholded decisions.
Step 3Compute metrics with the positive class and averaging convention explicitly defined.
Step 4Inspect a curve/confusion matrix/residual distribution in addition to scalar metrics.
Lesson summary

What to remember

  • ROC AUC evaluates a particular aspect of prediction quality. No single metric captures discrimination, calibration, threshold costs and subgroup performance simultaneously; choose metrics based on the decision and class/base-rate structure.
  • Generate predictions on held-out data.
  • Using a metric without defining the positive class, averaging rule, units or threshold that gives it meaning.
  • Recompute the metric from a tiny set of predictions/labels or residuals by hand.