Accuracy, flexibility and overfitting
The one idea the exam asks about every single year: more flexible models fit the training data better, but past a point they get worse on new data. Learn to score a model, split the data honestly, and explain the U-shape.
Format 3 of the exam (“discuss the flexibility / overfitting / bias and variance”) appeared in 2023, 2024 and 2025, each time about a different model. This stage teaches the general answer; every later stage asks you to apply it to one model.
Book: what to read for this stage
ISLP 2.2.1 Measuring the quality of fit and 2.2.2 The bias-variance trade-off: skim both. The slides only state the bias-variance decomposition; you don’t need its derivation. Read the figures 2.9 to 2.12 captions carefully: they are the U-shape. The validation-set idea is in 5.1.1 (one page), but the slides only use the idea, not the chapter.
1. Scoring a model: MSE and error rate
For regression, the mean squared error averages the squared misses:
\[\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^{n}\big(y_i - \hat f(x_i)\big)^2.\]
Squaring makes every miss positive and punishes big misses much more than small ones.
For classification, the error rate is the fraction of wrong labels:
\[\mathrm{ER} = \frac{1}{n}\sum_{i=1}^{n}\mathbb 1(\hat y_i \neq y_i),\]
where the indicator \(\mathbb 1(\cdot)\) is 1 when its statement is true and 0 otherwise. Accuracy is \(1 - \mathrm{ER}\).
Worked example. True \(y = (3, 5, 2, 8)\), predictions \(\hat y = (2.5, 5, 3, 7)\). Misses \(-0.5, 0, 1, -1\); squares \(0.25, 0, 1, 1\); sum \(2.25\); \(\mathrm{MSE} = 2.25/4 = 0.5625\).
Labels \((A, B, B, A, C)\) predicted as \((A, B, A, A, C)\): one wrong out of five, \(\mathrm{ER} = 0.2\).
2. Training error lies: train, validation and test
What we care about is how \(\hat f\) does on new data it has never seen, the test error. The training error (on the data used to fit) is usually too optimistic, and a flexible enough model can drive it to zero by memorising the noise.
So we split the data:
- Training set: fit the model (find its parameters).
- Validation set: choose the settings that training can’t choose, such as KNN’s \(K\), a polynomial’s degree, ridge’s \(\lambda\) or a tree’s pruning \(\alpha\). Pick the setting with the lowest validation error.
- Test set: used once, at the end, to report an honest estimate of performance.
The labs do exactly this with split_dataset(D, [0.6, 0.2, 0.2]). If you choose settings by looking at the test set, its error stops being honest: you have, in effect, trained on it.
You try ten values of \(K\) for KNN and pick the one with the lowest test error, then report that test error as your model’s performance. What’s wrong?
Any data used to make a decision is no longer “unseen”. Training error is even worse for choosing \(K\): it always picks \(K = 1\), whose training error is 0.
3. Flexibility, bias and variance
A model’s flexibility is how wiggly a shape it can fit. Every model has a knob that controls it: KNN’s \(K\), a polynomial’s degree, the number of parameters, ridge’s \(\lambda\), a tree’s number of leaves, a network’s number of hidden neurons.
The expected test MSE splits into three parts (slides, ISLP 2.7):
\[\mathbb E\big[(Y_0 - \hat f(X_0))^2\big] = \mathrm{bias}^2(\hat f) + \mathrm{variance}(\hat f) + \mathrm{Var}(\epsilon).\]
- Bias: error from the model’s assumptions being too simple (fitting a line to a curve). High for restrictive models.
- Variance: how much \(\hat f\) would change if you trained on a different sample. High for flexible models, which chase the noise in whichever sample they got.
- Irreducible error \(\mathrm{Var}(\epsilon)\): the floor.
As flexibility rises, bias falls and variance rises. So:
- Training error keeps falling as flexibility rises.
- Test error falls, then rises: a U-shape. The bottom of the U is the right amount of flexibility. Left of it: underfitting (too much bias). Right of it: overfitting (too much variance).
Worked example: the exam answer template. Every “discuss flexibility” question can be answered with this skeleton (fill in the knob):
“The model becomes more flexible as [knob] increases (for KNN: as \(K\) decreases). More flexibility lowers the bias, because the model can follow more complicated shapes, but raises the variance, because it starts to fit the noise in the particular training sample. Too little flexibility underfits; too much overfits, so training error keeps falling while test error rises again. We choose [knob] by minimising the error on a validation set (or by cross-validation), not on the training set.”
A model is made more and more flexible. Tick every statement that is true.
Test error is U-shaped, not monotone. The irreducible error \(\mathrm{Var}(\epsilon)\) doesn’t depend on the model at all.
4. When does a flexible model win?
The right amount of flexibility depends on the data (ISLP 2.4 exercise 1):
| Situation | Flexible model is… | Why |
|---|---|---|
| huge \(n\), small \(p\) | better | plenty of data to learn the shape without overfitting |
| huge \(p\), small \(n\) | worse | few points per parameter: it overfits |
| truth is highly non-linear | better | a restrictive model has high bias |
| \(\mathrm{Var}(\epsilon)\) very large | worse | there’s a lot of noise to (wrongly) fit |
You have only 30 observations, 25 predictors, and the noise is large. Which do you expect to have the lower test error?
Small \(n\), large \(p\) and large noise all punish flexibility: a flexible model would fit the noise (high variance). Training error would mislead you towards the flexible one.
Stage test
(5 marks, exam format 3.) Explain why choosing the model with the lowest training error is a bad idea, using the words flexibility, bias, variance and overfitting, and say how you would choose instead.
Training error always decreases as a model becomes more flexible, because a flexible model can bend to pass close to every training point; at the extreme (e.g. 1-NN, a tree with one point per leaf) it is zero. But flexibility trades bias for variance: a flexible model has low bias, since it can follow complicated shapes, and high variance, since it changes a lot with the particular sample and starts fitting the noise \(\epsilon\). That is overfitting, and it makes the test error rise again, so test error is U-shaped in flexibility while training error is not. The lowest training error therefore points to the most overfitted model. Instead, fit each candidate on the training set, choose the one with the lowest error on a separate validation set (or by cross-validation), and report its error on a test set used only once.
A classifier labels 20 test points and gets 15 right. What is its error rate?
\(5/20 = 0.25\).
Which part of the expected test error cannot be reduced by any model?
\(\epsilon\) is unpredictable from \(X\), so its variance is a floor for every model.
The data have a huge number of observations, few predictors, and a strongly curved true relationship. Which model would you expect to do better on test data?
Large \(n\) lets a flexible model learn the shape without overfitting, and a straight line would have high bias on a curved truth.
What is the validation set for?
Training set: fit. Validation set: choose. Test set: report, once.