Start hereCan a model look excellent because we accidentally gave it the answer?
Leakage occurs when training uses information that would not be available at prediction time or information from held-out data. It often produces impressive but unreproducible performance.
Building interactive view…
Technical lensFormalise what the visual is doing
Leakage can enter through preprocessing, target proxies, duplicates, grouped entities, temporal look-ahead or feature engineering performed before splitting.
Technical questionUse a tiny case to make the mechanism observable. Leakage can enter through preprocessing, target proxies, duplicates, grouped entities, temporal look-ahead or feature engineering performed before splitting. Verify one intermediate quantity, state change or mapping independently; then predict the consequence of this change: Fit preprocessing on the entire dataset, then compare the inflated validation score with a leakage-safe pipeline.
Practitioner lensUse it responsibly
Build transformations inside pipelines and audit every feature with one question: “Would I know this value at the moment of prediction?”
Transfer testTransfer this idea to a new example and justify each decision using this practitioner rule: Build transformations inside pipelines and audit every feature with one question: “Would I know this value at the moment of prediction?” Then explain what should change if you deliberately test: Fit preprocessing on the entire dataset, then compare the inflated validation score with a leakage-safe pipeline.
Worked explorationUse the visual as an experiment, not decoration
Predict loan default using a field created after default resolution. The offline score becomes excellent because the feature contains future information. Remove it and rebuild the pipeline using only information available at decision time.
Technical lens
Leakage can enter through preprocessing, target proxies, duplicates, grouped entities, temporal look-ahead or feature engineering performed before splitting.
Practitioner check
Build transformations inside pipelines and audit every feature with one question: “Would I know this value at the moment of prediction?”
Prediction before interactionFit preprocessing on the entire dataset, then compare the inflated validation score with a leakage-safe pipeline.
Exploration walkthroughTurn the interaction into an evidence trail
Predict loan default using a field created after default resolution. The offline score becomes excellent because the feature contains future information. Remove it and rebuild the pipeline using only information available at decision time. Before moving the control, state your prediction. After the visual changes, name the specific state, statistic, boundary or mapping that changed and explain why that change is consistent—or inconsistent—with your prediction.
- Record one observable quantity before the interaction and the same quantity afterwards.
- Change one factor at a time so the causal effect of the control is inspectable.
- Use an edge or failure case to discover where the concept stops behaving as the simple story suggests.
