Follow the transformation
Measure the problem by column, row group and time/segment.
Investigate the data-generating process before choosing a fix.
Choose deletion, correction, imputation, transformation or retention with an explicit reason.
Data Preparation for ML · Lesson 16
Missing Value Imputation
Missing Value Imputation is a data-quality decision, not merely a cleaning command.
What Missing Value Imputation actually means
Missing Value Imputation is a data-quality decision, not merely a cleaning command. The correct treatment depends on how the issue arose, whether it carries information, and how the treatment changes the population or downstream model.
Missing Value Imputation matters because models learn from the feature representation they receive, not from the raw concept in your head. Scaling, encoding, imputation and construction can change geometry and signal, and learned steps must stay inside validation folds.
Deeper walkthrough
Read Missing Value Imputation as a mechanism, not a recipe
Treat this as a sequence of observable decisions rather than one opaque command. Stage 1: Measure the problem by column, row group and time/segment. Stage 2: Investigate the data-generating process before choosing a fix. Stage 3: Choose deletion, correction, imputation, transformation or retention with an explicit reason. Final checkpoint: Record an indicator or audit trail when the fact that a value was missing/changed may itself matter.
Know what would convince you
- Fit the transformation on training data only and confirm validation/test rows are transformed without refitting.
- Inspect transformed shape, feature names/ranges and several rows against the raw inputs.
Useful distinctionDeletion: Removes affected rows/columns; can bias the population.
Click a stage to inspect what happens, what changes, and what should be checked before moving on.
Stage 1
Measure the problem by column
Measure the problem by column, row group and time/segment. For Missing Value Imputation, identify the exact state before this stage, the operation or rule applied here, and the observable state afterwards so the mechanism remains inspectable.
State focus: identify exactly what changed at this stage and what observable evidence confirms that change.
Trace the mechanism step by step
- Measure the problem by column, row group and time/segment.
- Investigate the data-generating process before choosing a fix.
- Choose deletion, correction, imputation, transformation or retention with an explicit reason.
- Fit learned preprocessing only on training data when predictive modelling is involved.
- Record an indicator or audit trail when the fact that a value was missing/changed may itself matter.
Make the concept concrete
Demonstration
Python / scikit-learn example
# Step 1 — Import the module so its functions/classes are available to the rest of this example.
import numpy as np
# Step 2 — Import only the named objects needed by the following steps, keeping dependencies explicit.
from sklearn.impute import SimpleImputer
# Step 3 — Construct `X` as an array so vectorised numerical operations can be applied consistently.
X=np.array([[1.],[np.nan],[3.]])
# Step 4 — Compute the right-hand expression and store its result in `imp` for the next step.
imp=SimpleImputer(strategy="median")
# Step 5 — Display the current value explicitly so the result/state can be inspected during execution.
print(imp.fit_transform(X).ravel())Expected / illustrative result
[1. 2. 3.] The median is learned from training data and reused unchanged on validation/test data.
Interpret the result.
For Missing Value Imputation, trace representative source rows/columns into the result and reconcile row counts, dtypes, keys or missing values that the operation could change.
Know what this is — and what it is not
DeletionRemoves affected rows/columns; can bias the population.Simple imputationUses a fixed statistic/category; easy but shrinks variability.Model-based imputationUses relationships with other variables; more assumptions and leakage risk.Missing indicatorPreserves information about whether a value was missing.When it is appropriate
Use Missing Value Imputation when the model/analysis requires a deliberate representation of raw features and the transformation can be fit without leaking future or held-out information.
When to stop or reconsider
Avoid transformations that are unnecessary for the chosen model, cannot be reproduced at inference time, or learn from data that should remain held out.
Failure modes to recognise
- Fitting scaling, encoding, imputation or feature selection on the full dataset before validation.
- Creating feature values whose meaning changes between training and inference.
- Ignoring output shape/feature names and losing track of what the transformed columns represent.
How to check the result
- Fit the transformation on training data only and confirm validation/test rows are transformed without refitting.
- Inspect transformed shape, feature names/ranges and several rows against the raw inputs.
- Place the transformation in a pipeline and cross-validate the complete workflow, not a preprocessed snapshot.
Demonstrate understanding
Try this:
Build a tiny, inspectable example of Missing Value Imputation. First measure the problem by column, row group and time/segment. Then investigate the data-generating process before choosing a fix. Write the expected result before running it, and explain one condition that would make the result misleading or invalid.
Use a tiny train/validation split. Fit only on train, print the transformed shape/values, and confirm validation transformation does not update learned state.
Check reasoning, not memorisation
Before trusting a result from Missing Value Imputation, which check provides the strongest evidence that you understand and applied it correctly?
Keep the important distinctions visible
Step 1Measure the problem by column, row group and time/segment.Step 2Investigate the data-generating process before choosing a fix.Step 3Choose deletion, correction, imputation, transformation or retention with an explicit reason.Step 4Fit learned preprocessing only on training data when predictive modelling is involved.What to remember
- Missing Value Imputation is a data-quality decision, not merely a cleaning command. The correct treatment depends on how the issue arose, whether it carries information, and how the treatment changes the population or downstream model.
- Measure the problem by column, row group and time/segment.
- Fitting scaling, encoding, imputation or feature selection on the full dataset before validation.
- Fit the transformation on training data only and confirm validation/test rows are transformed without refitting.