Start hereHow do we estimate generalisation when one split is too fragile?
Cross-validation rotates which observations are held out, giving several views of model performance and revealing variability that one split can hide.
Building interactive view…
Technical lensFormalise what the visual is doing
K-Fold assumes exchangeable independent observations; stratification preserves class proportions; grouped CV keeps related entities together; time-series CV preserves temporal order.
Technical questionUse a tiny case to make the mechanism observable. K-Fold assumes exchangeable independent observations; stratification preserves class proportions; grouped CV keeps related entities together; time-series CV preserves temporal order. Verify one intermediate quantity, state change or mapping independently; then predict the consequence of this change: Use ordinary K-Fold on repeated measurements from the same person and locate the leakage path.
Practitioner lensUse it responsibly
Choose the split strategy from the data-generating process, not from habit. Report fold variability as well as the mean.
Transfer testTransfer this idea to a new example and justify each decision using this practitioner rule: Choose the split strategy from the data-generating process, not from habit. Report fold variability as well as the mean. Then explain what should change if you deliberately test: Use ordinary K-Fold on repeated measurements from the same person and locate the leakage path.
Worked explorationUse the visual as an experiment, not decoration
Take 10 cases and perform 5-fold CV. Each case is validation once and training four times. Compare fold scores; their variation shows why a single holdout can be fragile.
Technical lens
K-Fold assumes exchangeable independent observations; stratification preserves class proportions; grouped CV keeps related entities together; time-series CV preserves temporal order.
Practitioner check
Choose the split strategy from the data-generating process, not from habit. Report fold variability as well as the mean.
Prediction before interactionUse ordinary K-Fold on repeated measurements from the same person and locate the leakage path.
Exploration walkthroughTurn the interaction into an evidence trail
Take 10 cases and perform 5-fold CV. Each case is validation once and training four times. Compare fold scores; their variation shows why a single holdout can be fragile. Before moving the control, state your prediction. After the visual changes, name the specific state, statistic, boundary or mapping that changed and explain why that change is consistent—or inconsistent—with your prediction.
- Record one observable quantity before the interaction and the same quantity afterwards.
- Change one factor at a time so the causal effect of the control is inspectable.
- Use an edge or failure case to discover where the concept stops behaving as the simple story suggests.
