HumachLearnLearn · explore · apply
Start learning
Math for ML · Lesson 12

Gradient Descent

Gradient Descent supplies mathematical intuition for how models represent predictions and learn from error.

ConceptWorked examplePracticeKnowledge check
Textbook walkthrough

What Gradient Descent actually means

Gradient Descent supplies mathematical intuition for how models represent predictions and learn from error. Vectors encode features/parameters, dot products create linear scores, loss functions quantify disagreement with targets, and gradients point toward local directions of change in differentiable objectives.

Gradient Descent matters because model predictions and optimisation are expressed through vectors, linear algebra, probability and loss functions. Connecting the mathematics to a concrete prediction or parameter update makes later algorithms much easier to understand.

Deeper walkthrough

Read Gradient Descent as a mechanism, not a recipe

Treat this as a sequence of observable decisions rather than one opaque command. Stage 1: Represent inputs/parameters as vectors or matrices. Stage 2: Compute the model score/prediction. Stage 3: Evaluate a loss against the observed target. Final checkpoint: Update parameters in a direction that reduces the objective, balancing fit and generalisation.

Mechanism

Follow the transformation

Represent inputs/parameters as vectors or matrices.

Compute the model score/prediction.

Evaluate a loss against the observed target.

Evidence

Know what would convince you

  • Write the dimensions and formula before substituting values.
  • Calculate a two- or three-value example by hand and compare every intermediate term.
Useful distinctionLoss: Objective contribution for prediction error.
Visual demonstration of Gradient Descent
Visual demonstration: use the diagram to trace the main objects and state changes involved in Gradient Descent.
Click a stage to inspect what happens, what changes, and what should be checked before moving on.
Stage 1

Represent inputs/parameters as vectors or matrices

Represent inputs/parameters as vectors or matrices. For Gradient Descent, identify the exact state before this stage, the operation or rule applied here, and the observable state afterwards so the mechanism remains inspectable.

State focus: identify exactly what changed at this stage and what observable evidence confirms that change.
Mathematical / formal view
Basic update: θ_(t+1) = θ_t - η ∇L(θ_t), where η is the learning rate and ∇L is the gradient.
How it works

Trace the mechanism step by step

  1. Represent inputs/parameters as vectors or matrices.
  2. Compute the model score/prediction.
  3. Evaluate a loss against the observed target.
  4. Compute or approximate how the loss changes with parameters.
  5. Update parameters in a direction that reduces the objective, balancing fit and generalisation.
Worked demonstration

Make the concept concrete

Demonstration

Python example

# Step 1 — Compute the right-hand expression and store its result in `theta` for the next step.
theta=0.0
# Step 2 — Compute the right-hand expression and store its result in `learning_rate` for the next step.
learning_rate=0.1
# L=(theta-3)^2 => gradient=2(theta-3)
# Step 3 — Iterate through the collection so the indented block is applied once for each item.
for step in range(5):
    # Step 4 — Compute the right-hand expression and store its result in `grad` for the next step.
    grad=2*(theta-3)
    # Step 5 — Execute this statement and inspect how it changes the current value, object or program state.
    theta -= learning_rate*grad
    # Step 6 — Display the current value explicitly so the result/state can be inspected during execution.
    print(step+1, round(theta,3))
Expected / illustrative result
theta moves 0 → 0.6 → 1.08 → 1.464 → ... toward the minimum at 3.
Interpret the result.

For Gradient Descent, connect the displayed result to the specific input and mechanism above; independently verify one value/state change rather than treating successful execution as proof.

Distinctions & related ideas

Know what this is — and what it is not

LossObjective contribution for prediction error.
GradientVector of partial derivatives of loss with respect to parameters.
Learning rateStep-size hyperparameter for gradient updates.
BiasSystematic error from restrictive assumptions.
VarianceSensitivity of the fitted model to training-sample variation.
Use deliberately

When it is appropriate

Use Gradient Descent when the mathematical object directly represents the model quantity or transformation being reasoned about and dimensions/units are explicit.

Boundary conditions

When to stop or reconsider

Stop if vectors/matrices, dimensions, signs, scales or probability assumptions are not defined; symbolic manipulation without those definitions can produce a formally valid but meaningless result.

Common mistakes

Failure modes to recognise

  • Combining quantities with incompatible dimensions/shapes or forgetting an intercept/normalisation term.
  • Interpreting coefficient or distance magnitude without considering feature scale.
  • Skipping a hand-computable case and therefore missing sign, axis or indexing errors.
Verification

How to check the result

  • Write the dimensions and formula before substituting values.
  • Calculate a two- or three-value example by hand and compare every intermediate term.
  • Perturb one input/parameter and predict the sign/direction of the result before recomputing it.
Hands-on practice

Demonstrate understanding

Try this:

Build a tiny, inspectable example of Gradient Descent. First represent inputs/parameters as vectors or matrices. Then compute the model score/prediction. Write the expected result before running it, and explain one condition that would make the result misleading or invalid.

Use two-dimensional vectors or a handful of probabilities so every arithmetic step fits on paper. Track shape, sign and units explicitly.
Knowledge check

Check reasoning, not memorisation

Before trusting a result from Gradient Descent, which check provides the strongest evidence that you understand and applied it correctly?

Quick reference

Keep the important distinctions visible

Step 1Represent inputs/parameters as vectors or matrices.
Step 2Compute the model score/prediction.
Step 3Evaluate a loss against the observed target.
Step 4Compute or approximate how the loss changes with parameters.
Lesson summary

What to remember

  • Gradient Descent supplies mathematical intuition for how models represent predictions and learn from error. Vectors encode features/parameters, dot products create linear scores, loss functions quantify disagreement with targets, and gradients point toward local directions of change in differentiable objectives.
  • Represent inputs/parameters as vectors or matrices.
  • Combining quantities with incompatible dimensions/shapes or forgetting an intercept/normalisation term.
  • Write the dimensions and formula before substituting values.