Ridge regression (regularisation)
Linear regression with a rule added: "keep the coefficients small unless the data really insists." It stops a model from overreacting to noise.
Question type: regression (and the same idea is used in week 3’s lab for logistic regression).
Week: 2.
Book: ISLP 6.2.1, 6.2.3, a bit of 6.1.3, and 6.2.2 is only named.
Techniques on this page: the penalty, the tuning dial \(\lambda\), standardising inputs, choosing \(\lambda\) with a validation set.
The problem it solves
Linear regression works brilliantly when you have many more rows than inputs. It goes wrong when:
- You have nearly as many inputs as rows (easy to reach once you add polynomial terms or many step columns). The fit memorises the data: near-zero training error, poor test error.
- Inputs copy each other (say \(X_3 = X_1 + X_2\)). Then the matrix formula can’t decide how to share credit between them, and the coefficients swing wildly from one sample to the next.
In both cases the problem is variance: the coefficients are jumpy. Ridge regression accepts being slightly wrong on average (a bit of bias) in exchange for being much steadier (a lot less variance). Often that lowers the test error a lot.
The idea in everyday words
Imagine coefficients cost money. Every unit of size you give a coefficient is “charged” to you. You’ll still give a coefficient a big value if it reduces the error a lot, but you won’t waste budget fitting tiny wiggles of noise. That charge is the penalty.
Words you’ll hear
| Word | Plain meaning |
|---|---|
| regularisation | adding a rule that discourages a model from being too complicated |
| shrinkage | pulling coefficients towards zero |
| penalty | the extra cost added to the error, here \(\lambda\sum\beta_j^2\) |
| \(\lambda\) (“lambda”), tuning parameter, hyperparameter | the dial that sets how strong the penalty is. You choose it; it isn’t learned from the training data |
| collinearity | inputs that (nearly) copy each other |
| standardise | rescale each input to average 0 and spread 1 |
| lasso | a cousin of ridge (named only): uses \(\sum|\beta_j|\) and can switch inputs off completely |
Technique 1: the penalty
Problem it solves: making “too-large coefficients” cost something.
\[\hat\beta^{\text{ridge}} = \text{the } \beta \text{ that makes }\ \underbrace{\sum_i\big(y_i - \beta_0 - \textstyle\sum_j\beta_jx_{ij}\big)^2}_{\text{RSS, the usual error}} \;+\; \underbrace{\lambda\sum_{j=1}^p\beta_j^2}_{\text{penalty}}\ \text{ as small as possible}\]
Notes:
- The penalty sums from \(j = 1\), so the intercept \(\beta_0\) is not penalised. It just sets the overall level; shrinking it would make no sense.
- It’s the squares of the coefficients, \(\beta_j^2\).
Slide typo: equation (60) should have the penalty squared, \(\lambda\sum_{j=1}^p\hat\beta_j^{2}\), starting at \(j = 1\).
Technique 2: the dial \(\lambda\)
Problem it solves: how much to shrink.
- \(\lambda = 0\): no penalty. Plain least squares.
- \(\lambda\) huge: every coefficient is pushed to almost 0. The model just predicts the average of \(y\).
- In between: coefficients shrink smoothly towards 0, but never exactly to 0.
Bigger \(\lambda\) means a stiffer model: bias goes up, variance goes down, training error goes up. Test error makes the familiar U shape in \(\lambda\) (the slides’ credit-card figure).
Worked example (one input, centred data). With centred data the model is \(\hat y = \beta x\), and the ridge goal is \(\sum(y_i - \beta x_i)^2 + \lambda\beta^2\). Setting its slope to zero (the usual trick) gives
\[\hat\beta^{\text{ridge}} = \frac{\sum x_iy_i}{\sum x_i^2 + \lambda}\]
Using the data from the linear regression page (\(\sum x_iy_i = 6\), \(\sum x_i^2 = 10\)):
| \(\lambda\) | 0 | 5 | 10 | 90 |
|---|---|---|---|---|
| \(\hat\beta\) | \(6/10 = 0.6\) | \(6/15 = 0.4\) | \(6/20 = 0.3\) | \(6/100 = 0.06\) |
It shrinks steadily but never hits zero. Compare the formula with plain least squares: the only change is “\(+\lambda\)” on the bottom.
Bonus (not on the slides, but useful): in matrix form the answer is \(\hat\beta^{\text{ridge}} = (\mathbf X^T\mathbf X + \lambda I)^{-1}\mathbf X^T\mathbf y\). Adding \(\lambda I\) (λ down the diagonal) makes the matrix invertible even when inputs copy each other or there are more inputs than rows. That’s the mathematical reason ridge fixes those problems.
Technique 3: standardise the inputs first
Problem it solves: the penalty is unfair otherwise.
A coefficient’s size depends on the units of its input. Measure income in pounds and its coefficient is tiny; measure it in thousands of pounds and the coefficient is 1000 times bigger, so the penalty hits it much harder, for no real reason. Plain least squares doesn’t care about units (the predictions come out the same), but ridge does. So rescale every input to average 0 and spread 1 before fitting.
Technique 4: choosing \(\lambda\) with a validation set
Problem it solves: you can’t choose \(\lambda\) by training error, because \(\lambda = 0\) always wins there (any penalty makes the training fit worse). That would defeat the point.
Split the data three ways (the lab’s split_dataset(D, [0.6, 0.2, 0.2])):
| Part | Used for |
|---|---|
| training (60%) | fit \(\hat\beta\) for each candidate \(\lambda\) |
| validation (20%) | pick the \(\lambda\) with the lowest error here |
| test (20%) | used once at the very end to report an honest score |
The same recipe chooses any setting you can’t learn from training data: polynomial degree, number of cut points, KNN’s \(K\). Try values on a log scale (0.001, 0.01, 0.1, 1, 10…), since what matters is the order of magnitude. (Cross-validation, a more efficient version, comes later in the course.)
Ridge vs lasso, in one line
Ridge shrinks every coefficient but keeps them all. The lasso uses \(\lambda\sum|\beta_j|\) instead and can set some coefficients to exactly 0, which also picks which inputs matter. The slides only name it.
What to read in the book
| Section | Read? | Why |
|---|---|---|
| 6.1.1, 6.1.2 Best subset, stepwise selection | Skip | Not on the slides |
| 6.1.3 Choosing the optimal model | Skim | Only the validation-set idea. Skip \(C_p\), AIC, BIC, adjusted \(R^2\) |
| 6.2.1 Ridge regression | Read | Core, including why to standardise |
| 6.2.2 The lasso | Skim | Only named on the slides |
| 6.2.3 Selecting the tuning parameter | Read | Short |
| 6.3, 6.4 | Skip | Not on the slides |
Practice prompt
I'm studying CS5100 Data Analysis (textbook ISLP, sections 6.1.3, 6.2.1-6.2.3). I want practice questions on RIDGE REGRESSION and choosing tuning parameters:
- when ordinary least squares struggles: n close to p, p > n, collinear predictors, and why that means high variance
- writing the ridge objective (RSS + lambda * sum of beta_j^2, intercept not penalised)
- computing one-predictor ridge estimates by hand with beta = Sxy / (Sxx + lambda) for several lambdas
- what happens to training RSS, bias, variance and test error as lambda increases (the U-shape)
- why predictors must be standardised before ridge but not before least squares
- why (X^T X + lambda I) is invertible even with collinearity
- choosing lambda (or degree, or K) with a train/validation/test split, and why the test set is used only once
- the difference between ridge and lasso in one or two sentences
Please:
1. Use plain, everyday language and explain any technical term in one short sentence.
2. Give me ONE question at a time, easy first, then harder, with small numbers I can do by hand.
3. Wait for my answer, then mark it and show the full working step by step.
4. If I get it wrong, explain the mistake and give me a similar question.
5. Every 5 questions, summarise my strengths and what to practise.