When reinforcement learning goes wrong
An optimiser does exactly what the reward says, not what you meant. Add tables that explode and a world that drifts away from training, and you have the three classic failures that motivate the rest of the module.
Kind: not an agent: the limits of everything on the week 4 pages.
Week: 4 (slides 51 to 56).
Book: not in Wooldridge.
Techniques on this page: spotting reward hacking, counting table sizes, recognising distribution shift.
The problem it solves
Value iteration and Q-learning find the policy that maximises expected return as defined by \(R\). That’s a strength and a danger: if \(R\) is only a proxy for the goal, the agent will find the gap. And everything this week was a table, which only works for tiny worlds. Knowing these limits is exam material, and it sets up topic 5 (function approximation) and topic 11 (specification gaming).
Words you’ll hear
| Word | Plain meaning |
|---|---|
| proxy reward | a measurable stand-in for what we really want |
| reward hacking | maximising the proxy in a way that defeats the intent |
| specification problem | the fault is in what we asked for, not in the optimiser |
| sample efficiency | how much experience learning needs |
| tabular | one stored number per state (or state–action pair) |
| distribution shift | the deployed agent meets states unlike its training ones |
| function approximation | computing \(V\), \(Q\) or \(\pi\) with a function (e.g. a neural network) instead of a table |
Technique 1: spotting reward hacking
Problem it solves: predicts how an agent will exploit a badly specified reward (slides 51 to 52).
- The boat race (Clark and Amodei 2016): rewarded for hitting score targets, the agent circled a lagoon hitting respawning targets for ever instead of finishing.
- RLHF (Gao et al. 2023): optimise harder against a learned reward model and the proxy keeps rising while true human preference eventually falls.
- The gridworld: give a positive reward per step and the “optimal” agent never reaches the goal.
There is no purely algorithmic fix. A better optimiser finds the loophole faster.
Worked example (a boat race MDP). Cells 0 → 1 → 2 → 3 → finish (+1). At cell 1 the boat can loop through a bonus lagoon (+0.3) and come back to cell 1. \(\gamma = 0.95\).
Looping for ever from cell 1 is worth \(0.3/(1-0.95) = 6\). Racing on is worth \(0.95^2 \times 1 \approx 0.90\). The optimal policy loops for ever.
Fix: pay the bonus only once. Now the state must record whether it’s been collected, \((cell, taken)\), because the reward depends on history: a Markov property lesson. The new optimal policy takes the bonus once, then finishes: worth \(0.95(0.3 + 0.95^3) = 1.10\) from the start.
Whose failure is it? (slide 52). The human wrote the imperfect reward, but if optimisers are known to exploit gaps, robustness to them is part of good algorithm design too. Mitigations: learn rewards from human comparisons (RLHF, itself a proxy), penalise side effects, limit how hard you optimise (RLHF’s KL penalty), monitor for “high score, weird behaviour”, keep a human in the loop.
Technique 2: counting table sizes
Problem it solves: shows why tabular methods don’t scale (slide 53).
\(n\) binary features give \(2^n\) states; a Q-table needs \(2^n \times |A|\) entries, each needing many visits.
Worked example. 10 features, 4 actions: \(1024 \times 4 = 4096\) entries. 20 features, 5 actions: \(2^{20} \times 5 = 5{,}242{,}880\). 30 features: over a billion states. A \(100 \times 100\) greyscale image as the state: \(256^{10000}\).
Worse, table entries share nothing: learning about one state teaches the agent nothing about a near-identical one. Function approximation (topic 5) computes \(Q(s, a)\) from features, so similar states share what’s learned.
Technique 3: recognising distribution shift
Problem it solves: explains why an agent that was fine in training misbehaves in deployment (slide 54).
\(Q\) and \(\pi\) are learned from the states the agent visited. Deployment can lead to states it rarely or never saw, where estimates are unreliable (in a table, maybe still the initial values). For LLM agents: fine-tuned on one mix of prompts and tools, it may behave unpredictably on another.
The takeaway (slide 56). MDPs formalise sequential decisions; basic RL learns values or policies from consequences when they can’t be supplied in advance. But it’s only as good as its state (Markov?), its observations (full?), its reward (the right one?) and its table (small enough?).
What to read
| Source | Read? | Why |
|---|---|---|
| Wooldridge | Not covered | |
| Amodei et al. (2016), “Concrete Problems in AI Safety” | Optional | readable; sections on reward hacking and distributional shift |
Practice prompt
I'm studying CS5940 Intelligent Agents, week 4 (MDPs and reinforcement learning). I want practice on THE LIMITS OF REINFORCEMENT LEARNING:
- defining reward hacking and predicting how an agent would exploit a described reward (boat race, gridworld with positive living reward, RLHF reward models)
- why reward hacking is a specification problem, and practical mitigations
- fixing a reward that depends on history by adding information to the state
- counting Q-table sizes for n binary features and |A| actions, and why tables don't generalise
- distribution shift, with examples for robots and LLM agents
- how these motivate function approximation
Please:
1. Use plain, everyday language and explain any symbol in one short sentence.
2. Give me ONE question at a time, easy first, then harder, with friendly numbers.
3. Wait for my answer, then mark it and show the full working.
4. If I get it wrong, explain the mistake and give me a similar question.
5. Every 5 questions, summarise my strengths and what to practise.