Follow the transformation
Scale features when units differ.
Choose k and a distance metric using validation.
Compute distances from a query to training cases.
Predictive Modelling · Lesson 60
KNN
K-nearest neighbours predicts from labelled training examples closest to the query under a chosen distance metric.
KNN
K-nearest neighbours predicts from labelled training examples closest to the query under a chosen distance metric. It stores the training data rather than learning a compact global equation, so feature scale, k and local sample density directly affect predictions.
Learning goal: explain why KNN behaves this way, apply it to a small example, and verify the result independently. Begin by being able to justify this first step: Scale features when units differ.
Deeper walkthrough
Read KNN as a mechanism, not a recipe
Treat this as a sequence of observable decisions rather than one opaque command. Stage 1: Scale features when units differ. Stage 2: Choose k and a distance metric using validation. Stage 3: Compute distances from a query to training cases. Final checkpoint: Inspect sensitivity to k and high-dimensional/noisy features.
Know what would convince you
- Confirm fitted transformations/models saw only training data.
- Retain fold/test predictions so metrics can be recomputed independently.
Useful distinctionTraining evidence: Information allowed to influence fitted state.
Click a stage to inspect what happens, what changes, and what should be checked before moving on.
Stage 1
Scale features when units differ
Scale features when units differ. For KNN, identify the exact state before this stage, the operation or rule applied here, and the observable state afterwards so the mechanism remains inspectable.
State focus: identify exactly what changed at this stage and what observable evidence confirms that change.
Trace the mechanism step by step
- Scale features when units differ.
- Choose k and a distance metric using validation.
- Compute distances from a query to training cases.
- Use neighbour voting for classification or averaging for regression.
- Inspect sensitivity to k and high-dimensional/noisy features.
Neighbour vote
Nearest labels to a query: [A, A, B], k=3 → prediction A.Expected / illustrative result
The result follows directly from the local neighbourhood; changing feature scaling can change which cases are nearest.
Interpret the result.
For KNN, connect the result to the fitted state, held-out data or prediction rule that produced it and independently check one prediction, split or metric component.
Place the concept correctly
Training evidenceInformation allowed to influence fitted state.Held-out evidenceIndependent observations used to estimate generalisation.InterpretationWhat the result supports, with assumptions and limitations.When it is appropriate
Use KNN when it answers a defined question in Predictive Modelling and its inputs/assumptions match the current data or program state.
When to stop or reconsider
Reconsider KNN when the required information is unavailable, the operation would violate a validation/data boundary, or a simpler operation answers the question more transparently.
Failure modes to recognise
- Learning preprocessing/feature/model choices from held-out test information.
- Comparing models under different splits or preprocessing and attributing the difference to the algorithm.
- Turning an association or model explanation into an unsupported causal claim.
How to check the result
- Confirm fitted transformations/models saw only training data.
- Retain fold/test predictions so metrics can be recomputed independently.
- Inspect errors/subgroups and compare with a baseline before generalising the conclusion.
Demonstrate understanding
Try this:
Construct a tiny example of KNN. First scale features when units differ. Then choose k and a distance metric using validation. Predict the result before execution and explain one boundary or failure case.
Use a tiny fixed split or synthetic example. State what is fitted, what remains held out, and what result you expect before running it.
Check reasoning, not memorisation
Which approach best demonstrates understanding of KNN?
Remember the logic
Step 1Scale features when units differ.Step 2Choose k and a distance metric using validation.Step 3Compute distances from a query to training cases.Step 4Use neighbour voting for classification or averaging for regression.What to remember
- K-nearest neighbours predicts from labelled training examples closest to the query under a chosen distance metric. It stores the training data rather than learning a compact global equation, so feature scale, k and local sample density directly affect predictions.
- Scale features when units differ.
- Learning preprocessing/feature/model choices from held-out test information.
- Confirm fitted transformations/models saw only training data.