Training, Validation & Model Selection

Hyperparameter Playground

See how model settings change training/validation behaviour and why hyperparameters should be selected without peeking at the test set.

14 · Pipeline 15 · Compare 16 · Tune 17 · Cross-validation 18 · Bias/variance 19 · History

Lab concept guide

What to observe while you experiment

Hyperparameters control model structure or training but are not learned directly from the fitted objective in the same way as model parameters. They should be selected with validation evidence while the final test set remains untouched.

MechanismChange one hyperparameter, refit on training data, and compare validation behaviour while holding the split and other settings fixed.

Failure modeChoosing settings from test performance or changing many hyperparameters at once and attributing the result to one of them.

VerificationRecord the fixed split/folds and compare candidate validation scores/variance under identical evaluation; open the test only after selection.

Experiment deliberately

Choose one complexity parameter. Predict its underfit/overfit direction, sweep it, and identify the validation-supported region before viewing test performance.

Three-way split:Tune on validation data first. Press “Evaluate final test” only after choosing a parameter.

DatasetModelParameterSeed

Ready.

Validation curve

Compare training and validation F1 as complexity/regularisation changes.

Chosen setting

The selected point is retrained on training + validation only when final test evaluation is requested.

Chosen-model separator

Replay how the selected classifier’s decision region is exposed without moving the observations.