Hyperparameter Optimisation
Hyperparameter optimisation searches configuration space for choices such as learning rate, tree depth, regularisation, kernel parameters and architecture size. The topic is split into focused lessons so definitions, implementation decisions, diagnostics and common failure modes can be learned separately.
Learn the mechanism one decision at a time
Work through the lessons in order if the topic is new. If you already know the basics, open the specific leaf lesson that matches the operation, diagnostic or failure mode you need.
1Definition→
2Mechanism→
3Example→
4Diagnostic→
5Decision
Manual and grid searchManual search is useful for intuition. Grid search is systematic but wastes trials when only a few dimensions strongly matter. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.
02Random searchRandom search samples independent configurations and often explores important dimensions more effectively than a full Cartesian grid for the same budget. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.
03Bayesian optimisation and TPESequential model-based methods use earlier trials to focus future samples on promising regions while retaining exploration. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.
04Successive Halving and HyperbandMulti-fidelity methods begin with many candidates at a small resource budget and allocate more epochs/data to strong candidates, reducing wasted training. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.
05Practical search designChoose a meaningful search space, log every trial, use early stopping when valid, and evaluate the final selected configuration on untouched data. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.