A4.3.6 · Machine learning approaches (HL only)
A4.3.6 · 机器学习方法(仅 HL)
Reinforcement Learning
强化学习
Nobody shows a warehouse robot a labelled map of the correct path to the charging dock. It tries a route, gets a small reward or penalty, and slowly works out which moves pay off, purely by doing.
没有人给仓库机器人一张标好正确路径的地图,让它直接走到充电桩。它尝试一条路线,得到一点小小的奖励或惩罚,然后慢慢摸索出哪些动作有回报,纯粹靠"做"来学习。
This is a third family alongside supervised and unsupervised learning: there is feedback (unlike unsupervised learning), but it is a reward signal for an action, not a labelled correct answer (unlike supervised learning).
这是与监督学习、无监督学习并列的第三个家族:它有反馈(这点不同于无监督学习),但那是针对一个动作的奖励信号,而不是一个带标签的正确答案(这点不同于监督学习)。
Explainer
讲解
Agent, environment, reward
智能体、环境、奖励
In reinforcement learning, an agent observes the current state of its environment, takes an action chosen by its policy, and receives a reward (positive or negative) that tells it how good that action was. What it is ultimately working toward is picking actions that build up the highest cumulative reward over time, not just the biggest reward on any single step.
在强化学习中,一个智能体观察其环境的当前状态,根据它的策略选择一个动作,并获得一个奖励(正的或负的),告诉它这个动作有多好。智能体的目标是选择能使其长期累积奖励最大化的动作,而不只是某一步的奖励。
| Term | In the warehouse robot example |
|---|---|
| Agent | The robot's control software |
| Environment | The warehouse floor, its shelves and the charging dock |
| State | The robot's current grid position |
| Action | Move up, down, left or right one cell |
| Reward | +10 for reaching the dock, −5 for hitting a shelf, −1 for every other move (to encourage a short route) |
| Policy | The rule mapping the robot's current state to the action it picks |
| 术语 | 在仓库机器人例子中 |
|---|---|
| 智能体 | 机器人的控制软件 |
| 环境 | 仓库地面、货架和充电桩 |
| 状态 | 机器人当前所在的网格位置 |
| 动作 | 向上、向下、向左或向右移动一格 |
| 奖励 | 到达充电桩 +10,撞到货架 −5,其余每一步 −1(以鼓励走更短的路线) |
| 策略 | 把机器人当前状态映射到它所选动作的规则 |
Explainer
讲解
Exploration vs exploitation
探索与利用
Exploitation means picking the action the agent currently believes is best. Exploration means trying something less certain, purely to learn whether it might actually be better. An agent that only exploits can get stuck repeating a mediocre route it found early; an agent that only explores never settles into using what it has learned.
利用指的是选择智能体目前认为最好的动作。探索指的是尝试一些不那么确定的动作,纯粹是为了了解它是否其实更好。一个只会利用的智能体可能会一直重复它早期找到的一条平庸路线;一个只会探索的智能体则永远无法安定下来使用它已经学到的东西。
Most of the time (probability 1 − ε) the agent exploits, picking its current best action. A small fraction of the time (probability ε) it explores instead, picking a random action. ε is often reduced over time as the agent gains confidence in what it has learned.
大多数时候(概率 1 − ε)智能体进行利用,选择它当前认为最好的动作。有一小部分时间(概率 ε)它转而探索,选择一个随机动作。随着智能体对已学到的东西越来越有信心,ε 通常会逐渐减小。
Explore
探索
A warehouse robot learns its route
仓库机器人学习路线
The robot starts top-left. Click a direction to move it. Hitting a shelf (hatched) costs −5 and bounces it back; reaching the dock (blue) gives +10; every other move costs −1. Watch the cumulative reward as you try a route.
机器人从左上角出发。点击一个方向来移动它。撞到货架(斜纹)扣 −5 并弹回原位;到达充电桩(蓝色)得 +10;其余每一步扣 −1。试着走一条路线,看看累积奖励如何变化。
The shortest route (right, right, right, down, down) reaches the dock in 5 moves for a total reward of 6 (+10 minus 4 steps of −1). A wandering route reaches the same dock with a lower total, exactly what the agent's policy is trying to avoid learning.
最短路线(右、右、右、下、下)用 5 步到达充电桩,总奖励为 6(+10 减去 4 步各 −1)。一条绕远路的路线到达同一个充电桩,总奖励会更低,而这正是智能体的策略要学着避免的。
Check your understanding
检验你的理解
A solar panel adjusts its own tilt angle throughout the day, trying different angles and measuring the electricity generated, to learn the best angle for each time of day. Why does this suit reinforcement learning rather than supervised learning?
一块太阳能板在一天中不断调整自己的倾角,尝试不同的角度并测量发电量,以学习每个时段的最佳角度。为什么这适合用强化学习而不是监督学习?
Common mistake
常见错误
RecurringDescribing a reward as if it were a label, "the reward tells the agent the correct action". A supervised label states the right answer directly; a reward only scores the action actually taken, and does not say what the best alternative would have been. The agent has to try many actions over many episodes, and compare their rewards, before a good policy emerges. It never sees the correct answer handed to it.
反复出现把奖励说成好像是一个标签,"奖励告诉智能体正确的动作"。监督学习的标签直接说明正确答案;而奖励只是对实际采取的那个动作打分,并不会说明本来最好的选择是什么。智能体必须在许多回合中尝试许多动作、比较它们的奖励,一个好的策略才会逐渐浮现。它从来不会被直接告知正确答案。
Wrong: "The reward acts like a label, telling the robot the correct direction to move at each step."
错误: "奖励就像一个标签,告诉机器人每一步应该往哪个正确方向移动。"
Right: "The reward only scores the move actually made; it does not state which move would have been best. The robot has to try many routes across many attempts and compare the rewards to gradually learn a policy that favours higher-reward actions."
正确: "奖励只给实际做出的那个动作打分;它并不会说明哪个动作本来最好。机器人必须在多次尝试中走过许多条路线,并比较奖励,才能逐渐学到一个偏向高奖励动作的策略。"
Revision questions
复习题
Specimen-paper style. Write an answer, then reveal the marking points and mark yourself.
采用样卷风格。写下答案,然后展开评分要点给自己打分。
[3]Identify the agent, the environment and the reward signal for a robot vacuum cleaner.
[3]为一台扫地机器人指出智能体、环境和奖励信号。
Marking points
评分要点
- Agent: the vacuum's control software/its decision-making system.
- Environment: the room, furniture and floor it navigates.
- Reward: positive for dirt collected or efficient cleaning, negative for collisions or getting stuck.
- 智能体:扫地机器人的控制软件 / 决策系统。
- 环境:它所行走的房间、家具和地面。
- 奖励:清扫到灰尘或高效清扫为正,碰撞或卡住为负。
[3]Explain the trade-off a reinforcement learning agent faces between exploring and exploiting.
[3]解释强化学习中探索与利用之间的权衡。
Marking points
评分要点
- Exploitation: choosing the action currently believed to give the highest reward.
- Exploration: trying a less certain action to discover whether it is actually better.
- Too much exploitation risks settling for a mediocre policy; too much exploration wastes reward on actions already known to be poor.
- 利用:选择当前认为奖励最高的动作。
- 探索:尝试一个不那么确定的动作,以了解它是否其实更好。
- 利用过多有可能满足于一个平庸的策略;探索过多则会在已知效果不佳的动作上浪费奖励。
[2]Define cumulative reward.
[2]定义累积奖励。
Marking points
评分要点
- The total reward an agent accumulates over time / across a series of actions, not the reward from a single step.
- This is what the agent's policy is trying to maximise.
- 智能体随时间 / 在一系列动作中累积的总奖励,而不是单独一步的奖励。
- 这正是智能体的策略要努力最大化的东西。
[4]Justify why a stock-trading bot would use reinforcement learning rather than supervised learning.
[4]论证为什么一个股票交易机器人会使用强化学习而不是监督学习。
Marking points
评分要点
- There is no fixed labelled data set of "correct" trades; the right decision depends on constantly changing market conditions.
- The bot can only measure an outcome (profit or loss) after acting, which is a reward signal, not a label.
- Reinforcement learning lets the bot keep adapting its policy as it interacts with a changing environment, which a fixed supervised model cannot do without being retrained on new labelled data.
- 不存在一个固定的、标好"正确"交易的数据集;正确的决策取决于不断变化的市场状况。
- 机器人只能在行动之后测量一个结果(盈利或亏损),这是一个奖励信号,而不是标签。
- 强化学习让机器人在与不断变化的环境互动时持续调整自己的策略,而一个固定的监督模型若不在新的带标签数据上重新训练,就做不到这一点。
Flashcards
记忆卡片
Active recall for this statement’s key terms.
针对本条考纲陈述关键术语的主动回忆练习。
Keep this statement’s practice with you
把这条考纲陈述的练习保存下来
Free account: save your progress and unlock every revision question for A4.3.6, and every other IB CS 2027 statement.
免费账户:解锁 A4.3.6 以及所有其他 IB CS 2027 考纲陈述的每一道复习题,并同步你的学习进度。