HumachLearnLearn · explore · apply
Start learning
Math for ML · Lesson 13

Bias and Variance

Bias and Variance supplies mathematical intuition for how models represent predictions and learn from error.

ConceptWorked examplePracticeKnowledge check
Textbook walkthrough

What Bias and Variance actually means

Bias and Variance supplies mathematical intuition for how models represent predictions and learn from error. Vectors encode features/parameters, dot products create linear scores, loss functions quantify disagreement with targets, and gradients point toward local directions of change in differentiable objectives.

Bias and Variance matters because model predictions and optimisation are expressed through vectors, linear algebra, probability and loss functions. Connecting the mathematics to a concrete prediction or parameter update makes later algorithms much easier to understand.

Deeper walkthrough

Read Bias and Variance as a mechanism, not a recipe

Treat this as a sequence of observable decisions rather than one opaque command. Stage 1: Represent inputs/parameters as vectors or matrices. Stage 2: Compute the model score/prediction. Stage 3: Evaluate a loss against the observed target. Final checkpoint: Update parameters in a direction that reduces the objective, balancing fit and generalisation.

Mechanism

Follow the transformation

Represent inputs/parameters as vectors or matrices.

Compute the model score/prediction.

Evaluate a loss against the observed target.

Evidence

Know what would convince you

  • Write the dimensions and formula before substituting values.
  • Calculate a two- or three-value example by hand and compare every intermediate term.
Useful distinctionLoss: Objective contribution for prediction error.
How it works

Trace the mechanism step by step

  1. Represent inputs/parameters as vectors or matrices.
  2. Compute the model score/prediction.
  3. Evaluate a loss against the observed target.
  4. Compute or approximate how the loss changes with parameters.
  5. Update parameters in a direction that reduces the objective, balancing fit and generalisation.
Worked demonstration

Make the concept concrete

Demonstration

Text example

Very simple model: training error high, validation error high → underfitting/high bias.
Very complex model: training error near zero, validation error much higher → overfitting/high variance.
Useful complexity: both errors low with a modest gap.
Expected / illustrative result
Learning curves make the bias–variance trade-off observable rather than purely theoretical.
Interpret the result.

For Bias and Variance, connect the displayed result to the specific input and mechanism above; independently verify one value/state change rather than treating successful execution as proof.

Distinctions & related ideas

Know what this is — and what it is not

LossObjective contribution for prediction error.
GradientVector of partial derivatives of loss with respect to parameters.
Learning rateStep-size hyperparameter for gradient updates.
BiasSystematic error from restrictive assumptions.
VarianceSensitivity of the fitted model to training-sample variation.
Use deliberately

When it is appropriate

Use Bias and Variance when the mathematical object directly represents the model quantity or transformation being reasoned about and dimensions/units are explicit.

Boundary conditions

When to stop or reconsider

Stop if vectors/matrices, dimensions, signs, scales or probability assumptions are not defined; symbolic manipulation without those definitions can produce a formally valid but meaningless result.

Common mistakes

Failure modes to recognise

  • Combining quantities with incompatible dimensions/shapes or forgetting an intercept/normalisation term.
  • Interpreting coefficient or distance magnitude without considering feature scale.
  • Skipping a hand-computable case and therefore missing sign, axis or indexing errors.
Verification

How to check the result

  • Write the dimensions and formula before substituting values.
  • Calculate a two- or three-value example by hand and compare every intermediate term.
  • Perturb one input/parameter and predict the sign/direction of the result before recomputing it.
Hands-on practice

Demonstrate understanding

Try this:

Build a tiny, inspectable example of Bias and Variance. First represent inputs/parameters as vectors or matrices. Then compute the model score/prediction. Write the expected result before running it, and explain one condition that would make the result misleading or invalid.

Use two-dimensional vectors or a handful of probabilities so every arithmetic step fits on paper. Track shape, sign and units explicitly.
Knowledge check

Check reasoning, not memorisation

Before trusting a result from Bias and Variance, which check provides the strongest evidence that you understand and applied it correctly?

Quick reference

Keep the important distinctions visible

Step 1Represent inputs/parameters as vectors or matrices.
Step 2Compute the model score/prediction.
Step 3Evaluate a loss against the observed target.
Step 4Compute or approximate how the loss changes with parameters.
Lesson summary

What to remember

  • Bias and Variance supplies mathematical intuition for how models represent predictions and learn from error. Vectors encode features/parameters, dot products create linear scores, loss functions quantify disagreement with targets, and gradients point toward local directions of change in differentiable objectives.
  • Represent inputs/parameters as vectors or matrices.
  • Combining quantities with incompatible dimensions/shapes or forgetting an intercept/normalisation term.
  • Write the dimensions and formula before substituting values.