HumachLearnLearn · explore · apply
Start learning
Hyperparameter Optimisation · Lesson 66

Bayesian Optimisation

Bayesian Optimisation is a hyperparameter-optimisation strategy.

ConceptWorked examplePracticeKnowledge check
Textbook walkthrough

What Bayesian Optimisation actually means

Bayesian Optimisation is a hyperparameter-optimisation strategy. Hyperparameters control the learning procedure rather than being estimated directly by the model fit, so tuning must be nested inside a validation design that protects the final evaluation data.

Bayesian Optimisation matters because hyperparameter optimisation is itself a data-driven selection process. Search spaces, budgets and adaptive choices must use validation evidence while leaving final test data untouched.

Deeper walkthrough

Read Bayesian Optimisation as a mechanism, not a recipe

Treat this as a sequence of observable decisions rather than one opaque command. Stage 1: Define a validation objective and one or more constraints/secondary metrics. Stage 2: Specify a plausible search space using domain knowledge and log scales where appropriate. Stage 3: Evaluate candidate configurations through cross-validation or a validation set. Final checkpoint: Refit the selected configuration on the full development data and evaluate once on the held-out test set.

Mechanism

Follow the transformation

Define a validation objective and one or more constraints/secondary metrics.

Specify a plausible search space using domain knowledge and log scales where appropriate.

Evaluate candidate configurations through cross-validation or a validation set.

Evidence

Know what would convince you

  • Log every candidate, score, budget and fold definition so the best configuration can be reproduced.
  • Compare the selected model with a reasonable default/baseline on the same validation protocol.
Useful distinctionGrid search: Exhaustive combinations on a fixed grid; expensive in many dimensions.
Visual demonstration of Bayesian Optimisation
Visual demonstration: use the diagram to trace the main objects and state changes involved in Bayesian Optimisation.
Click a stage to inspect what happens, what changes, and what should be checked before moving on.
Stage 1

Define a validation objective and one…

Define a validation objective and one or more constraints/secondary metrics. At this stage of Bayesian Optimisation, keep the incoming data or object separate from the learned parameter, transformed object, or statistic so the change can be reproduced and independently checked.

Transformation focus: keep the input and produced parameters/result separate so the change is observable and reproducible.
How it works

Trace the mechanism step by step

  1. Define a validation objective and one or more constraints/secondary metrics.
  2. Specify a plausible search space using domain knowledge and log scales where appropriate.
  3. Evaluate candidate configurations through cross-validation or a validation set.
  4. Use adaptive methods only on validation information.
  5. Refit the selected configuration on the full development data and evaluate once on the held-out test set.
Worked demonstration

Make the concept concrete

Demonstration

Text example

Trial 1: learning_rate=0.01 → validation score 0.79
Trial 2: learning_rate=0.10 → 0.84
Trial 3 is chosen using a model of promising regions rather than uniformly at random.
Expected / illustrative result
Adaptive optimisation trades exploration of uncertain regions against exploitation of regions that already look good.
Interpret the result.

For Bayesian Optimisation, connect the reported result to the exact training/validation/prediction step that produced it and check one prediction, fold or metric component independently.

Distinctions & related ideas

Know what this is — and what it is not

Grid searchExhaustive combinations on a fixed grid; expensive in many dimensions.
Random searchSamples configurations; often more efficient when only some dimensions matter.
Bayesian/TPEUses results from previous trials to choose promising new configurations.
Successive halving/HyperbandAllocates small resources broadly and more resources to promising candidates.
Use deliberately

When it is appropriate

Use Bayesian Optimisation when there is a defined validation objective, a defensible search space and enough compute/data to compare candidate configurations fairly.

Boundary conditions

When to stop or reconsider

Stop expanding the search when validation noise, budget or an ill-defined metric dominates; a larger search can overfit the validation process itself.

Common mistakes

Failure modes to recognise

  • Searching implausible parameter ranges without understanding which parameters control model capacity/regularisation.
  • Using the test set during search or repeatedly peeking at it between search rounds.
  • Comparing search methods with different budgets or fold assignments and attributing differences to the algorithm alone.
Verification

How to check the result

  • Log every candidate, score, budget and fold definition so the best configuration can be reproduced.
  • Compare the selected model with a reasonable default/baseline on the same validation protocol.
  • Evaluate the final selected configuration once on untouched test data and report the search/selection uncertainty.
Hands-on practice

Demonstrate understanding

Try this:

Build a tiny, inspectable example of Bayesian Optimisation. First define a validation objective and one or more constraints/secondary metrics. Then specify a plausible search space using domain knowledge and log scales where appropriate. Write the expected result before running it, and explain one condition that would make the result misleading or invalid.

Use a tiny, bounded search space and fixed folds first. Write why each range is plausible and compare the winner with a default baseline under the same budget.
Knowledge check

Check reasoning, not memorisation

Before trusting a result from Bayesian Optimisation, which check provides the strongest evidence that you understand and applied it correctly?

Quick reference

Keep the important distinctions visible

Step 1Define a validation objective and one or more constraints/secondary metrics.
Step 2Specify a plausible search space using domain knowledge and log scales where appropriate.
Step 3Evaluate candidate configurations through cross-validation or a validation set.
Step 4Use adaptive methods only on validation information.
Lesson summary

What to remember

  • Bayesian Optimisation is a hyperparameter-optimisation strategy. Hyperparameters control the learning procedure rather than being estimated directly by the model fit, so tuning must be nested inside a validation design that protects the final evaluation data.
  • Define a validation objective and one or more constraints/secondary metrics.
  • Searching implausible parameter ranges without understanding which parameters control model capacity/regularisation.
  • Log every candidate, score, budget and fold definition so the best configuration can be reproduced.