A4.2.2 · Data preprocessing (HL only)
The Role of Feature Selection
Every extra column of data handed to a model is a bet that it's actually worth including. Feature selection is how that bet gets made deliberately, rather than by just throwing every available column at the model and hoping for the best.
Explainer
Feature selection is the process used to identify and retain the most informative attributes of a data set, while removing the ones that are redundant or irrelevant. It's a genuine balancing act: removing a feature that actually mattered hurts the model just as much as keeping one that only adds noise.
A feature here just means one column, one measurable attribute, of the data being fed into the model. A model predicting exam results might be handed a student's attendance rate, hours of sleep, number of practice papers completed, and even their favourite subject, but not all of those are actually useful for the prediction.
Worked example
Three strategies, one prediction task
A fitness app wants to predict how many calories a workout burned, using logged features: workout duration, average heart rate, body weight, outdoor temperature, day of the week, and shirt colour. Click each strategy to see how it would actually be applied to this exact problem.
Check your understanding
A team repeatedly trains a full machine learning model on different subsets of the available input features, comparing the resulting accuracy each time to decide which subset to keep. Which feature selection strategy is this?
Common mistake
RecurringA genuinely documented pattern: candidates treat feature selection as a minor, optional step, assuming a more sophisticated model or algorithm matters more to a project's success than the quality of the features it's actually given. In practice, the quality of the chosen features is frequently the bigger factor: a simple model with well-chosen features regularly outperforms a sophisticated model buried in irrelevant ones.
Treats model choice as the priority: "If the calorie-prediction model isn't accurate enough, switch to a more powerful algorithm." Skips past the possibility that irrelevant features, like shirt colour, are actively confusing whichever algorithm gets used.
Treats feature quality as the priority: "Before trying a more powerful algorithm, check whether irrelevant features like shirt colour are still in the data set, since no algorithm, however sophisticated, can be expected to ignore noise on its own." Correctly identifies feature quality as often the bigger lever, not just the choice of model.
Revision questions
Past-paper style. Write an answer, then reveal the marking points and mark yourself.
[2]Define the term feature selection.
Marking points
评分要点
- Feature selection is choosing a subset of the available input features to use in a model.
- It keeps the informative features and drops those that are irrelevant or redundant.
- 特征选择是从可用的输入特征中选择一个子集用于模型。
- 它保留有信息量的特征,舍弃无关或冗余的特征。
[4]Distinguish between filter methods and wrapper methods, referring to computational cost in your answer.
Marking points
评分要点
- A filter method ranks each feature by a statistical score, such as its correlation with the target, independently of any model.
- A wrapper method trains and evaluates the actual model on different feature subsets and keeps the best-performing subset.
- The filter is computationally cheap because it scores each feature once.
- The wrapper is far more computationally expensive because it retrains the model many times, once per subset tried.
- 过滤法用统计得分对每个特征排序,如它与目标的相关性,与任何模型无关。
- 包裹法在不同的特征子集上训练并评估实际模型,保留表现最好的子集。
- 过滤法计算代价低,因为它对每个特征只打一次分。
- 包裹法计算代价高得多,因为它多次重新训练模型,每尝试一个子集就训练一次。
[6]A hospital is building a model to predict patient readmission risk, using features including age, blood pressure, number of past admissions, and the patient's assigned hospital bed number. Describe how each of the three feature selection strategies could be applied to this data set.
Marking points
评分要点
- Filter method: score each feature against readmission, for example correlation or a statistical test; the bed number would score near zero and be dropped, age and past admissions would score highly and be kept.
- Wrapper method: train the readmission model on candidate subsets (for example add or remove one feature at a time) and keep whichever subset gives the best validation performance.
- Embedded method: use a model that performs selection during training, for example a penalty that shrinks the weight of the bed-number feature toward zero as it fits.
- In all three the bed number, having no real relationship to readmission, should end up excluded, while age, blood pressure and past admissions are retained.
- 过滤法:把每个特征与再入院做打分,例如相关性或统计检验;床位号得分接近零而被舍弃,年龄和既往入院次数得分高而被保留。
- 包裹法:在候选子集上训练再入院模型(例如每次加入或移除一个特征),保留在验证集上表现最好的子集。
- 嵌入法:使用一个在训练中执行选择的模型,例如一个在拟合时把床位号特征权重压向零的惩罚项。
- 三种方法中,床位号因与再入院没有真实关系,最终都应被排除,而年龄、血压和既往入院次数被保留。
Flashcards
Active recall for this statement’s key terms.
Keep this statement’s practice with you
Free account: unlock every revision question and flashcard for A4.2.2, and every other IB CS 2027 statement.