Follow the transformation
Evaluate candidate splits on training data.
Choose the split with the best impurity/error improvement.
Repeat recursively in child nodes.
Predictive Modelling · Lesson 61
Decision Trees
A decision tree predicts by recursively splitting the feature space into regions.
Decision Trees
A decision tree predicts by recursively splitting the feature space into regions. At each node it chooses a feature and threshold/category rule that reduces impurity for classification or prediction error for regression, then continues until a stopping or pruning condition is reached.
Learning goal: explain why Decision Trees behaves this way, apply it to a small example, and verify the result independently. Begin by being able to justify this first step: Evaluate candidate splits on training data.
Deeper walkthrough
Read Decision Trees as a mechanism, not a recipe
Treat this as a sequence of observable decisions rather than one opaque command. Stage 1: Evaluate candidate splits on training data. Stage 2: Choose the split with the best impurity/error improvement. Stage 3: Repeat recursively in child nodes. Final checkpoint: Validate because deep trees can fit noise very closely.
Know what would convince you
- Confirm fitted transformations/models saw only training data.
- Retain fold/test predictions so metrics can be recomputed independently.
Useful distinctionTraining evidence: Information allowed to influence fitted state.
Click a stage to inspect what happens, what changes, and what should be checked before moving on.
Stage 1
Evaluate candidate splits on training data
Evaluate candidate splits on training data. At this stage of Decision Trees, keep the incoming data or object separate from the learned parameter, transformed object, or statistic so the change can be reproduced and independently checked.
Transformation focus: keep the input and produced parameters/result separate so the change is observable and reproducible.
Trace the mechanism step by step
- Evaluate candidate splits on training data.
- Choose the split with the best impurity/error improvement.
- Repeat recursively in child nodes.
- Control depth/minimum samples or prune to limit variance.
- Validate because deep trees can fit noise very closely.
One split
If age < 30 → class A; otherwise → class B.Expected / illustrative result
A tree is a sequence of explicit decision rules; a real fitted tree chooses thresholds from data.
Interpret the result.
For Decision Trees, connect the result to the fitted state, held-out data or prediction rule that produced it and independently check one prediction, split or metric component.
Place the concept correctly
Training evidenceInformation allowed to influence fitted state.Held-out evidenceIndependent observations used to estimate generalisation.InterpretationWhat the result supports, with assumptions and limitations.When it is appropriate
Use Decision Trees when it answers a defined question in Predictive Modelling and its inputs/assumptions match the current data or program state.
When to stop or reconsider
Reconsider Decision Trees when the required information is unavailable, the operation would violate a validation/data boundary, or a simpler operation answers the question more transparently.
Failure modes to recognise
- Learning preprocessing/feature/model choices from held-out test information.
- Comparing models under different splits or preprocessing and attributing the difference to the algorithm.
- Turning an association or model explanation into an unsupported causal claim.
How to check the result
- Confirm fitted transformations/models saw only training data.
- Retain fold/test predictions so metrics can be recomputed independently.
- Inspect errors/subgroups and compare with a baseline before generalising the conclusion.
Demonstrate understanding
Try this:
Construct a tiny example of Decision Trees. First evaluate candidate splits on training data. Then choose the split with the best impurity/error improvement. Predict the result before execution and explain one boundary or failure case.
Use a tiny fixed split or synthetic example. State what is fitted, what remains held out, and what result you expect before running it.
Check reasoning, not memorisation
Which approach best demonstrates understanding of Decision Trees?
Remember the logic
Step 1Evaluate candidate splits on training data.Step 2Choose the split with the best impurity/error improvement.Step 3Repeat recursively in child nodes.Step 4Control depth/minimum samples or prune to limit variance.What to remember
- A decision tree predicts by recursively splitting the feature space into regions. At each node it chooses a feature and threshold/category rule that reduces impurity for classification or prediction error for regression, then continues until a stopping or pruning condition is reached.
- Evaluate candidate splits on training data.
- Learning preprocessing/feature/model choices from held-out test information.
- Confirm fitted transformations/models saw only training data.