HumachLearnLearn · explore · apply
Start learning
Evaluation, Robustness & Advanced ML

Train–Test Distribution Shift Simulator

Hold the trained classifier fixed while covariate shift or concept drift changes the evaluation environment.

20 · Metrics21 · Thresholds22 · Explain23 · Robustness24 · Shift25 · Time series
Lab concept guide

What to observe while you experiment

Distribution shift asks what happens when deployment/evaluation data no longer follow the training environment. Covariate shift changes input distribution; concept drift changes the relationship between inputs and target, so a fixed model can fail even when its code is unchanged.

MechanismTrain once, hold the model fixed, then change only the evaluation distribution so performance change can be attributed to shift rather than retraining.
Failure modeRetraining automatically during the experiment or conflating input drift with a changed target relationship.
VerificationCompare train and shifted feature distributions plus performance/calibration; verify that model parameters remain fixed while the evaluation environment changes.
Experiment deliberately
Apply covariate shift and concept drift separately. Predict which plots/metrics should move for each type before running the simulation.
Deployment data can change. The model is trained once on the blue training distribution; only the evaluation environment changes as shift severity increases.
Ready.

Training distribution, separator and shifted test data

Performance versus shift severity