Applied experimentation

Challenge Mode

Solve model-selection goals using validation evidence, receive scored feedback, and open the final test only after the goal is met.

Prediction sandbox Code ↔ visual Challenge mode 36 concept challenges

Lab concept guide

What to observe while you experiment

Model selection is a decision process under uncertainty. The challenge is to use validation evidence to choose a workflow while keeping the final test set sealed, so success reflects generalisation rather than repeated test-set feedback.

MechanismUse training data to fit, validation evidence to compare choices, and reveal the test set only after the selection rule is satisfied.

Failure modeOptimising choices after repeatedly seeing test performance, which converts the test set into another validation set.

VerificationRecord the decision rule and chosen configuration before opening the test result; check whether the same choice would have been made without test information.

Experiment deliberately

Complete one challenge twice with different validation evidence. Explain why your chosen model/hyperparameters changed—or why they should not.

ChallengeModelFeature mappingScalingKNN kTree depth

Ready.

Development geometry

The validation set is visible; the final test set remains hidden during model selection.

Attempt history

Scoring discipline: you can iterate on training and validation. The final test becomes available only after the stated validation target and generalisation-gap criterion are both satisfied.