A4.2.3 · Data preprocessing (HL only)

The Importance of Dimensionality Reduction

It's tempting to think a model given more information will always make better predictions. Past a certain point, the opposite happens: extra features actively make a model worse, a genuine problem with a name of its own.

HL only Typically 4–6 marks · AO1/AO2 Command term: describe

Explainer

Each feature a model is given adds another dimension to the space it's trying to find patterns in. The curse of dimensionality describes the range of genuine problems that show up once a data set has too many dimensions relative to how much data it actually contains.

A school wants to build a model that flags which students are at risk of falling behind, starting with three sensible features: attendance rate, average grade, and number of missed assignments. Someone suggests adding fifty more, locker number, favourite lunch item, hair colour, bus route, in the hope that more detail means a better model. It's the opposite: this is exactly the scenario each problem below plays out against.

Worked example

Seven ways more dimensions hurts

Click each one to see exactly how it shows up once the school's model balloons from 3 features to over 50.

Explainer

Reducing dimensions, keeping what matters

Dimensionality reduction means reducing the number of variables in a data set while preserving the relevant aspects of the data, cutting the trivial 50-feature version back down toward something closer to the original three that actually mattered, without throwing away genuinely useful signal in the process.

Deciding which dimensions to cut can be done manually, using judgement about what's actually likely to matter, or with statistical tools built for exactly this. Two such tools you may see named, principal component analysis (PCA) and linear discriminant analysis (LDA), are beyond the scope of this course, you're not expected to know how either one works, only that dimensionality reduction as a goal exists and why it matters.

Worked example

Reducing dimensions with scikit-learn

You're not expected to know the maths behind a tool like this, only that it exists and roughly what it does: take a data set with a large number of columns and compress it down to a much smaller number of new columns that still capture most of what mattered in the original. This uses scikit-learn, a real, widely used Python library for exactly this kind of task.

reduce_dimensions.pyRun locally, e.g. in VS Code

import numpy as np
from sklearn.decomposition import PCA

# A simplified version of the school's data set: 5 students, 6 already-scaled features
students = np.array([
    [0.92, 0.20, 0.10, 0.30, 0.88, 0.40],
    [0.61, 0.85, 0.40, 0.10, 0.55, 0.20],
    [0.78, 0.45, 0.20, 0.20, 0.70, 0.30],
    [0.45, 0.95, 0.60, 0.05, 0.40, 0.10],
    [0.88, 0.25, 0.10, 0.30, 0.82, 0.40],
])

print("Original shape:", students.shape)

pca = PCA(n_components=2)
reduced = pca.fit_transform(students)

print("Reduced shape:", reduced.shape)
print(np.round(reduced, 2))

Running this prints Original shape: (5, 6) then Reduced shape: (5, 2): six columns per student compressed down to two, chosen by the algorithm to preserve as much of the meaningful variation between students as it can. This isn't wired into this page's own in-browser console: scikit-learn needs roughly 80 MB of supporting libraries to run, far too much to download just for one page, so try this one in your own Python setup instead (pip install scikit-learn numpy).

Check your understanding

After being trained on the school's 50-feature version of the data set, the model correctly flags every at-risk student it was trained on, but performs poorly on the following year's new students. Which problem is this?

Common mistake

A recurring assumption flagged directly for this exact statement

RecurringA genuinely documented pattern: candidates assume dimensionality reduction automatically makes a model better, treating "fewer dimensions" as an unconditional improvement. It isn't. The actual goal is simplifying the data while keeping what's genuinely useful in it, and cutting the wrong dimension can throw away information the model actually needed, making it worse, not better.

Treats it as an automatic improvement: "Reducing the number of features will make the model more accurate." States a possible outcome as if it were guaranteed, when a badly chosen reduction can remove a feature the model genuinely relied on.

✓

States the actual trade-off: "Reducing the number of features can simplify the model and reduce overfitting, but only if the removed features weren't actually informative; cutting a genuinely useful feature would make predictions worse, not better." Correctly frames it as a balance to strike, not a guaranteed win.

Revision questions

Past-paper style. Write an answer, then reveal the marking points and mark yourself.

Question 1 · 2 marks

[2]Define the term curse of dimensionality.

Question 2 · 4 marks

[4]Explain how adding more dimensions to a data set affects data sparsity, and why that matters for a model's predictions.

Question 3 · 6 marks

[6]A hospital's patient-readmission model has grown from 5 features to 80. Describe three specific problems this growth could cause, referring to the curse of dimensionality.

Flashcards

Active recall for this statement’s key terms.