Follow the transformation
Measure the problem by column, row group and time/segment.
Investigate the data-generating process before choosing a fix.
Choose deletion, correction, imputation, transformation or retention with an explicit reason.
Data Cleaning & Missing Data · Lesson 46
Category Standardisation
Category Standardisation is a data-quality decision, not merely a cleaning command.
What Category Standardisation actually means
Category Standardisation is a data-quality decision, not merely a cleaning command. The correct treatment depends on how the issue arose, whether it carries information, and how the treatment changes the population or downstream model.
Category Standardisation matters because missing, duplicated, inconsistent or extreme values are evidence about the data-generating process, not merely inconveniences to delete. Treatment choices can change populations and conclusions.
Deeper walkthrough
Read Category Standardisation as a mechanism, not a recipe
Treat this as a sequence of observable decisions rather than one opaque command. Stage 1: Measure the problem by column, row group and time/segment. Stage 2: Investigate the data-generating process before choosing a fix. Stage 3: Choose deletion, correction, imputation, transformation or retention with an explicit reason. Final checkpoint: Record an indicator or audit trail when the fact that a value was missing/changed may itself matter.
Know what would convince you
- Compare missingness, distributions, row counts and key constraints before and after the change.
- Inspect representative changed rows and confirm the rule with domain/data documentation.
Useful distinctionDeletion: Removes affected rows/columns; can bias the population.
Trace the mechanism step by step
- Measure the problem by column, row group and time/segment.
- Investigate the data-generating process before choosing a fix.
- Choose deletion, correction, imputation, transformation or retention with an explicit reason.
- Fit learned preprocessing only on training data when predictive modelling is involved.
- Record an indicator or audit trail when the fact that a value was missing/changed may itself matter.
Make the concept concrete
Demonstration
Python example
# Step 1 — Compute the right-hand expression and store its result in `raw` for the next step.
raw = ["UK", "uk", "United Kingdom", "U.K."]
# Step 2 — Compute the right-hand expression and store its result in `map_` for the next step.
map_ = {"uk":"United Kingdom","united kingdom":"United Kingdom","u.k.":"United Kingdom"}
# Step 3 — Compute the right-hand expression and store its result in `clean` for the next step.
clean = [map_.get(x.strip().lower(), x) for x in raw]
# Step 4 — Display the current value explicitly so the result/state can be inspected during execution.
print(clean)Expected / illustrative result
['United Kingdom', 'United Kingdom', 'United Kingdom', 'United Kingdom']
Interpret the result.
For Category Standardisation, trace representative source rows/columns into the result and reconcile row counts, dtypes, keys or missing values that the operation could change.
Know what this is — and what it is not
DeletionRemoves affected rows/columns; can bias the population.Simple imputationUses a fixed statistic/category; easy but shrinks variability.Model-based imputationUses relationships with other variables; more assumptions and leakage risk.Missing indicatorPreserves information about whether a value was missing.When it is appropriate
Use Category Standardisation when it helps diagnose, document or correct a data-quality issue without destroying information needed for the downstream question.
When to stop or reconsider
Do not “clean” automatically when the apparent anomaly may carry signal, reflect data collection, or require domain adjudication; preserve an audit trail of changes.
Failure modes to recognise
- Treating missing values, outliers or labels as purely technical defects without investigating how they were generated.
- Applying learned cleaning/imputation using information from validation/test data.
- Changing values without recording which rows changed and how distributions/counts were affected.
How to check the result
- Compare missingness, distributions, row counts and key constraints before and after the change.
- Inspect representative changed rows and confirm the rule with domain/data documentation.
- In predictive work, fit learned cleaning only on training data and verify the pipeline reproduces that boundary.
Demonstrate understanding
Try this:
Build a tiny, inspectable example of Category Standardisation. First measure the problem by column, row group and time/segment. Then investigate the data-generating process before choosing a fix. Write the expected result before running it, and explain one condition that would make the result misleading or invalid.
Start by measuring the problem, not fixing it. Keep a before/after table of counts or distributions and inspect the exact records affected by the rule.
Check reasoning, not memorisation
Before trusting a result from Category Standardisation, which check provides the strongest evidence that you understand and applied it correctly?
Keep the important distinctions visible
Step 1Measure the problem by column, row group and time/segment.Step 2Investigate the data-generating process before choosing a fix.Step 3Choose deletion, correction, imputation, transformation or retention with an explicit reason.Step 4Fit learned preprocessing only on training data when predictive modelling is involved.What to remember
- Category Standardisation is a data-quality decision, not merely a cleaning command. The correct treatment depends on how the issue arose, whether it carries information, and how the treatment changes the population or downstream model.
- Measure the problem by column, row group and time/segment.
- Treating missing values, outliers or labels as purely technical defects without investigating how they were generated.
- Compare missingness, distributions, row counts and key constraints before and after the change.