Follow the transformation
Define the prediction unit and what “new” means: new row, subject, group or future time.
Reserve test data for final evaluation when feasible.
Use cross-validation inside development to compare models/hyperparameters.
Validation & Leakage · Lesson 60
Nested Cv
Nested Cv is a model-validation design.
What Nested Cv actually means
Nested Cv is a model-validation design. Validation estimates how a trained procedure will generalise to new cases, so the split must reproduce the independence structure and time/entity boundaries expected at deployment.
Nested Cv matters because validation estimates future performance only when held-out examples are genuinely independent under the deployment scenario. Random, stratified, grouped and temporal splits answer different generalisation questions.
Deeper walkthrough
Read Nested Cv as a mechanism, not a recipe
Treat this as a sequence of observable decisions rather than one opaque command. Stage 1: Define the prediction unit and what “new” means: new row, subject, group or future time. Stage 2: Reserve test data for final evaluation when feasible. Stage 3: Use cross-validation inside development to compare models/hyperparameters. Final checkpoint: Use group- or time-aware splits when ordinary random splitting would leak related/future information.
Know what would convince you
- Write down the prediction unit and which records are allowed to coexist across train/validation/test before splitting.
- Inspect fold/group/time indices directly and assert that forbidden overlap is zero.
Useful distinctionHoldout: One train/test split; simple but higher variance.
Trace the mechanism step by step
- Define the prediction unit and what “new” means: new row, subject, group or future time.
- Reserve test data for final evaluation when feasible.
- Use cross-validation inside development to compare models/hyperparameters.
- Keep all preprocessing, feature selection and tuning inside each training fold.
- Use group- or time-aware splits when ordinary random splitting would leak related/future information.
Make the concept concrete
Demonstration
Python / scikit-learn example
# Step 1 — Import only the named objects needed by the following steps, keeping dependencies explicit.
from sklearn.model_selection import KFold
# Step 2 — Compute the right-hand expression and store its result in `X` for the next step.
X = list(range(6))
# Step 3 — Iterate through the collection so the indented block is applied once for each item.
for fold,(tr,va) in enumerate(KFold(3,shuffle=True,random_state=1).split(X),1):
# Step 4 — Display the current value explicitly so the result/state can be inspected during execution.
print(fold, "train", tr.tolist(), "valid", va.tolist())Expected / illustrative result
Each sample is validation exactly once across three folds; metrics are aggregated across folds.
Interpret the result.
For Nested Cv, connect the reported result to the exact training/validation/prediction step that produced it and check one prediction, fold or metric component independently.
Know what this is — and what it is not
HoldoutOne train/test split; simple but higher variance.K-foldEach fold serves once as validation; assumes rows are exchangeable.StratifiedPreserves class proportions approximately.Group K-foldKeeps all rows from the same group together.Time-series splitTrains on past and validates on later data.Nested CVOuter loop estimates generalisation; inner loop selects/tunes.When it is appropriate
Use Nested Cv when it reflects how genuinely unseen cases will arrive and keeps every learned choice inside the training portion of each evaluation split.
When to stop or reconsider
Choose a different split strategy when observations share subjects/groups, have temporal order, or otherwise violate independent random splitting assumptions.
Failure modes to recognise
- Letting the final test set influence feature engineering, model choice, tuning or threshold selection.
- Splitting related groups or future/past records in a way that leaks information across folds.
- Reporting one lucky split without examining variability or preserving the exact split logic.
How to check the result
- Write down the prediction unit and which records are allowed to coexist across train/validation/test before splitting.
- Inspect fold/group/time indices directly and assert that forbidden overlap is zero.
- Run preprocessing/tuning inside each training fold and reserve the final test set for one final evaluation.
Demonstrate understanding
Try this:
Build a tiny, inspectable example of Nested Cv. First define the prediction unit and what “new” means: new row, subject, group or future time. Then reserve test data for final evaluation when feasible. Write the expected result before running it, and explain one condition that would make the result misleading or invalid.
Draw the rows/groups/times as blocks and label exactly which block trains, validates and tests each step. Check for any information path crossing the boundary.
Check reasoning, not memorisation
Before trusting a result from Nested Cv, which check provides the strongest evidence that you understand and applied it correctly?
Keep the important distinctions visible
Step 1Define the prediction unit and what “new” means: new row, subject, group or future time.Step 2Reserve test data for final evaluation when feasible.Step 3Use cross-validation inside development to compare models/hyperparameters.Step 4Keep all preprocessing, feature selection and tuning inside each training fold.What to remember
- Nested Cv is a model-validation design. Validation estimates how a trained procedure will generalise to new cases, so the split must reproduce the independence structure and time/entity boundaries expected at deployment.
- Define the prediction unit and what “new” means: new row, subject, group or future time.
- Letting the final test set influence feature engineering, model choice, tuning or threshold selection.
- Write down the prediction unit and which records are allowed to coexist across train/validation/test before splitting.