First exams 2027 · HL only

A4.2.1 · Data preprocessing (HL only)

The Significance of Data Cleaning

A machine learning model can only ever be as good as the data it learns from. Before any training happens, that data almost always needs real work done to it first, and skipping that work is one of the most common ways a model quietly ends up wrong.

HL only Typically 4–6 marks · AO1/AO2 Command term: describe

Explainer

There's an old computing proverb for this: garbage in, garbage out. A model trained on flawed data doesn't just produce slightly worse results, it can actively learn the wrong patterns, and no amount of clever algorithm design afterward can fully undo that.

Df

Data cleaning is the process of finding and correcting problems in a data set, outliers, duplicates, errors, irrelevant entries, inconsistent formatting and missing values, before that data is used to train a model. Clean data produces a model that's more accurate on data it has already seen, and more importantly, generalises better to genuinely new data it hasn't.

Worked example

Six cleaning techniques

Each technique tackles a genuinely different kind of problem. Click each one to see how it's detected and fixed, with a fresh example.

Try it yourself

Detecting an outlier with numpy

Real data cleaning is usually done in code, not by eye. This runs the same Z-score method described above on a small screen-time data set, one entry, a typo, sticks out clearly once the numbers are actually calculated.

main.py
Loading Python engine…
Output
Press Run to see the output here.

Running in offline fallback mode. This example needs numpy, which only the full Python engine (Pyodide) provides, so it isn't available here. If Pyodide can't load, usually a blocked school network or no internet connection, pressing Run will show a clear message explaining why, rather than a confusing crash.

The Z-score for the mistyped "400" comes out well above the threshold of 3, exactly the kind of number that gets a value flagged for a human to check, without anyone having to spot it by eye first.

Explainer

Normalization vs standardization

Once the data is clean, one more preprocessing step usually matters: putting every feature on a comparable scale, since a feature measured in the thousands can otherwise dominate a model over one measured in single digits, regardless of which one is actually more important.

Df

Normalization rescales a feature into a fixed range, typically 0 to 1, so every feature contributes on the same footing regardless of its original units.

Standardization rescales a feature around its own average, so the result centres on zero with a spread of exactly one standard deviation, which suits data with outliers or no fixed natural minimum and maximum better than squeezing it into a fixed range would.

Try it yourself

Normalizing and standardizing with numpy

The same five numbers, transformed two different ways. Notice normalization always lands between 0 and 1, while standardization spreads out around zero, some values negative, some positive.

main.py
Loading Python engine…
Output
Press Run to see the output here.

Running in offline fallback mode. This example needs numpy, which only the full Python engine (Pyodide) provides, so it isn't available here. If Pyodide can't load, usually a blocked school network or no internet connection, pressing Run will show a clear message explaining why, rather than a confusing crash.

Try changing data to include a genuine outlier (like the previous example's 400) and re-running: normalization squashes every other value into a tiny sliver near 0, while standardization keeps the ordinary values much more spread out and readable, exactly why standardization suits data with outliers better.

Check your understanding

A data set has one feature measured in millimetres (values from 0 to 5,000) and another measured as a percentage (values from 0 to 100). Before training, which preprocessing step directly addresses this mismatch in scale?

Common mistake

A recurring gap flagged directly for these two specific techniques

RecurringA genuinely documented pattern: candidates either skip normalization and standardization entirely when describing preprocessing, or use the two terms interchangeably as if they were the same operation. They solve related but genuinely different problems, and picking the wrong one, or applying it inconsistently between training and later data, can quietly hurt a model's performance.

✕

Treats them as the same step: "The data was normalized to standardize its scale." Uses both words as if they described one identical operation, missing that a fixed 0-to-1 range and a mean-0/standard-deviation-1 distribution are two different transformations suited to different situations.

✓

Names the actual difference: "The data was normalized to rescale it into a fixed 0-to-1 range, since the algorithm being used is sensitive to feature scale; a data set with major outliers would have used standardization instead, since squeezing an outlier into a fixed range distorts everything else." Correctly treats them as two distinct tools chosen for different reasons.

Revision questions

Past-paper style. Write an answer, then reveal the marking points and mark yourself.

Question 1 · 2 marks

[2]Define the term outlier, and state one statistical method used to detect one.

Question 2 · 3 marks

[3]Distinguish between normalization and standardization, referring to when each is more appropriate.

Question 3 · 6 marks

[6]A weather station's data set has several missing temperature readings, some clearly impossible values (like 300°C), and dates recorded in three different formats. Describe how each of these three issues should be addressed before the data is used to train a model.

Flashcards

Active recall for this statement’s key terms.