Simple linear regression
Fit the closest straight line to the data. You'll derive the formula yourself from stage 2's "set the slope to zero", fit a line by hand, and score it with RSS, R² and RSE.
The 2024 paper’s first question was linear regression: why you’d use it (with an example), the model formula, the RSS formula, how degrees of freedom relate to flexibility, and a small calculation. This stage and the next two cover every part.
Book: what to read for this stage
ISLP 3.1.1 (least squares, the \(\hat\beta\) formulas) and 3.1.3 (RSE and \(R^2\)): read. 3.1.2: skim only the idea that \(\hat\beta\) changes from sample to sample; skip standard errors, confidence intervals, t-statistics and p-values (not on the slides).
Slide typos
Eq. (4) writes \(\sum(y_i - \hat\beta_0 + \hat\beta_1x_i)^2\): it should be \(\sum\big(y_i - (\hat\beta_0 + \hat\beta_1x_i)\big)^2\). Eq. (15)’s denominator must be squared, \(\sum(x_i - \bar x)^2\). The TSS sum should start at \(i = 1\).
1. The model and least squares
One input \(X\), a numeric output \(Y\):
\[Y = \beta_0 + \beta_1X + \epsilon.\]
\(\beta_0\) is the intercept (the line’s value at \(X = 0\)), \(\beta_1\) the slope (the change in \(Y\) per unit increase in \(X\)). The exam writes this \(f(X) = \lambda_0 + \lambda_1X\) with \(\lambda \in \mathbb R^2\).
Why use it? It’s simple, fast, interpretable (the slope says how strongly and in which direction \(Y\) moves with \(X\)), it needs little data, and many fancier methods are extensions of it. Example: predicting a house’s price from its floor area.
Given estimates \(\hat\beta_0, \hat\beta_1\), the residuals are the vertical misses \(e_i = y_i - \hat y_i\), and least squares chooses the line that minimises the residual sum of squares:
\[\mathrm{RSS}(\beta_0, \beta_1) = \sum_{i=1}^n\big(y_i - \beta_0 - \beta_1x_i\big)^2.\]
Which is the residual sum of squares for the model \(f(X) = \lambda_0 + \lambda_1X\) and data \(D = \{(X_n, Y_n)\}_{n=1}^N\)?
Square each vertical miss \(Y_n - f(X_n)\) and add them up. The second option is the slide-typo version (wrong sign); the last is the TSS.
2. Deriving the formulas
The RSS is a bowl-shaped (convex) function of \((\beta_0, \beta_1)\) with a single lowest point. Find it by setting both partial derivatives to zero (stage 2, chain rule):
\[\frac{\partial\mathrm{RSS}}{\partial\beta_0} = -2\sum_i(y_i - \beta_0 - \beta_1x_i) = 0, \qquad \frac{\partial\mathrm{RSS}}{\partial\beta_1} = -2\sum_i x_i(y_i - \beta_0 - \beta_1x_i) = 0.\]
The first gives \(\sum y_i = n\beta_0 + \beta_1\sum x_i\), so
\[\boxed{\hat\beta_0 = \bar y - \hat\beta_1\bar x}\]
(the line always passes through \((\bar x, \bar y)\)). Substituting into the second and tidying up gives
\[\boxed{\hat\beta_1 = \frac{\sum_i(x_i - \bar x)(y_i - \bar y)}{\sum_i(x_i - \bar x)^2} = \frac{S_{xy}}{S_{xx}}}.\]
Two consequences of the first equation: the residuals sum to zero, \(\sum_i e_i = 0\), whenever there’s an intercept; and \(\hat\beta_1 = r_{xy}\,s_y/s_x\), the correlation rescaled into units of \(Y\) per unit of \(X\).
Worked example. \(x = (1, 2, 3, 4, 5)\), \(y = (2, 4, 5, 4, 5)\).
- Means \(\bar x = 3\), \(\bar y = 4\).
- Centred: \(x - \bar x = (-2, -1, 0, 1, 2)\), \(y - \bar y = (-2, 0, 1, 0, 1)\).
- \(S_{xy} = 4 + 0 + 0 + 0 + 2 = 6\), \(S_{xx} = 4 + 1 + 0 + 1 + 4 = 10\).
- \(\hat\beta_1 = 0.6\), \(\hat\beta_0 = 4 - 0.6 \times 3 = 2.2\). Line: \(\hat y = 2.2 + 0.6x\).
- Fitted \((2.8, 3.4, 4.0, 4.6, 5.2)\); residuals \((-0.8, 0.6, 1.0, -0.6, -0.2)\), which sum to 0.
- \(\mathrm{RSS} = 0.64 + 0.36 + 1 + 0.36 + 0.04 = 2.4\).
Why do the residuals of a least-squares line with an intercept always sum to zero?
It’s an algebraic fact about the fitted line, not a probability statement: it holds for every data set.
3. How good is the fit? TSS, RSS and \(R^2\)
- TSS \(= \sum_i(y_i - \bar y)^2\): the variation in \(Y\) before regression (the RSS of the flat line \(\hat y = \bar y\)).
- RSS: the variation left over after regression.
- \[R^2 = 1 - \frac{\mathrm{RSS}}{\mathrm{TSS}} \in [0, 1]:\] the proportion of the variance of \(Y\) explained by the model. It has no units.
\(R^2\) near 1: the line explains most of the variation. Near 0: it explains little, either because the noise is large or because the true relationship isn’t a line. In simple regression \(R^2 = r_{xy}^2\), the squared correlation.
Worked example (continued). \(\mathrm{TSS} = 4 + 0 + 1 + 0 + 1 = 6\), \(\mathrm{RSS} = 2.4\), so \(R^2 = 1 - 2.4/6 = 0.6\). The correlation of this data is \(0.7746\), and \(0.7746^2 = 0.6\). ✓
4. RSE and degrees of freedom
The residual standard error estimates the typical size of \(\epsilon\), in the units of \(Y\):
\[\mathrm{RSE} = \sqrt{\frac{\mathrm{RSS}}{n - p - 1}}\]
with \(p\) inputs (\(p = 1\) here, so \(n - 2\)). Why not divide by \(n\)? Each estimated parameter “uses up” one degree of freedom: the residuals must satisfy \(p + 1\) equations (the derivative conditions), so only \(n - p - 1\) of them are free. Same reason the sample variance divides by \(n - 1\): its deviations must sum to zero.
Degrees of freedom and flexibility. A model’s number of free parameters (its degrees of freedom) measures its flexibility. A line has 2. More parameters, more flexible: lower bias, higher variance. With \(n\) points, a model with \(n\) parameters can pass through every point (RSS \(= 0\), nothing left to estimate the noise with): perfectly overfitted. That’s the link the 2024 paper asked about.
Also worth knowing: \(\hat\beta\) is a random variable (another sample gives another line). In simple regression \(\mathrm{var}(\hat\beta_1) = \sigma^2/S_{xx}\): estimates are steadier with less noise, more data and more spread-out \(x\) values.
Worked example. From the running example, \(\mathrm{RSE} = \sqrt{2.4/(5 - 2)} = \sqrt{0.8} \approx 0.894\). A model with \(n = 20\), \(p = 3\) and \(\mathrm{RSS} = 34\): \(\mathrm{RSE} = \sqrt{34/16} \approx 1.458\).
A simple linear regression on \(n = 4\) points has \(\mathrm{RSS} = 4.2\). Compute the RSE.
\(\sqrt{4.2/(4 - 2)} = \sqrt{2.1} \approx 1.449\).
Model A predicts sales (in thousands) with RSE 2.1. Model B predicts house prices (in pounds) with RSE 21,000. Which fits better?
RSE is in units of \(Y\). \(R^2\) is unit-free, so it’s the fair comparison.
Stage test
(4 marks, exam formats 1 and 2; 2024 Q1.) Why would you use a linear regression model? Include a real-world example, and write the model for one input with parameters \(\lambda \in \mathbb R^2\).
Linear regression is used when the output is a real number and we expect it to change roughly linearly with the inputs. It is simple, cheap to fit, needs relatively little data and is easy to interpret: the slope says how much \(Y\) changes, on average, per unit increase in \(X\), so it is useful for both prediction and inference. Example: predicting a flat’s monthly rent (output, in pounds) from its floor area in square metres (input). With one input the model is \(f(X) = \lambda_0 + \lambda_1X\), \(\lambda = (\lambda_0, \lambda_1) \in \mathbb R^2\), where \(\lambda_0\) is the intercept and \(\lambda_1\) the slope; \(\lambda\) is chosen to minimise the residual sum of squares \(\sum_n(Y_n - f(X_n))^2\) on the training data.
A fit with \(n = 12\) points and \(p = 1\) input has \(\mathrm{RSS} = 90\). Compute the RSE.
\(\sqrt{90/(12 - 1 - 1)} = \sqrt{9} = 3\).
A least-squares line has \(R^2 = 0.36\). What is the correlation between \(x\) and \(y\), if the slope is negative?
In simple regression \(R^2 = r^2\), so \(|r| = 0.6\), and \(r\) has the sign of the slope.
You fit polynomials of increasing degree to 10 data points. What happens at degree 9?
As many parameters as data points means a perfect fit to the training data, noise included. Training RSS 0 says nothing good about the test error.