Linear regression
Draw the straight line (or flat surface) that runs closest to the data, then use it to predict. The foundation most later models are built on.
Question type: regression (the answer is a number).
Week: 2.
Book: ISLP 3.1, 3.2.1, 3.3.1, 3.3.2 (interactions), parts of 3.3.3. See the reading table at the bottom.
Techniques on this page: least squares, solving for the best line by hand, the matrix formula, gradient descent, judging the fit (RSE and \(R^2\)), dummy variables, interactions, checking the assumptions.
The problem it solves
You have inputs (TV advertising budget) and a numeric answer (sales). You want two things: predict sales for a new budget, and understand how much each extra pound of advertising adds. Linear regression assumes the simplest possible shape, a straight line, so the whole rule is described by a couple of numbers you can read directly.
Why bother with something so simple? Because it’s easy to fit, easy to read, and nearly every later model is a tweak of it: polynomial regression bends it, ridge regression calms it down, and logistic regression squashes it into a probability.
The model
One input (simple linear regression):
\[Y = \beta_0 + \beta_1 X + \epsilon\]
- \(\beta_0\) (“beta zero”), the intercept: the value of the line when \(X = 0\).
- \(\beta_1\), the slope: how much \(Y\) goes up for each extra 1 unit of \(X\).
- \(\epsilon\): the luck the line can’t explain.
Several inputs (multiple linear regression):
\[Y = \beta_0 + \beta_1X_1 + \beta_2X_2 + \dots + \beta_pX_p + \epsilon\]
Each \(\beta_j\) means: the average change in \(Y\) when \(X_j\) goes up by 1, with all the other inputs held fixed.
Words you’ll hear
| Word | Plain meaning |
|---|---|
| coefficients, \(\beta\) | the numbers that set the line’s position and tilt |
| \(\hat\beta\) | our estimate of those numbers from data |
| fitted value, \(\hat y_i\) | what the line predicts for training row \(i\) |
| residual, \(e_i = y_i - \hat y_i\) | how far the real point is above or below the line |
| RSS | residual sum of squares: add up the squared residuals. The line’s total “wrongness” |
| least squares | choosing the line with the smallest RSS |
| design matrix | the data grid with an extra column of 1s added at the front |
| learning rate, \(\eta\) (“eta”) | the step size in gradient descent |
| epoch | one full step of gradient descent |
Technique 1: least squares (what “closest line” means)
Problem it solves: there are infinitely many lines. We need a rule for which one is best.
For each point, measure the vertical gap to the line (the residual), square it, and add them all up:
\[\text{RSS}(\beta_0, \beta_1) = \sum_{i=1}^n \big(y_i - (\beta_0 + \beta_1 x_i)\big)^2\]
The best line is the one with the smallest RSS. Squaring stops positive and negative gaps cancelling out and punishes big misses more.
Slide typo: slide equation (4) is missing brackets. It must be \(y_i - (\hat\beta_0 + \hat\beta_1 x_i)\), not \(y_i - \hat\beta_0 + \hat\beta_1 x_i\).
Technique 2: finding the best line exactly, by setting the slope to zero
Problem it solves: “smallest RSS” is a goal. This technique gives an actual answer.
The RSS, viewed as a function of \(\beta_0\) and \(\beta_1\), is shaped like a bowl (the slides show it as a 3D surface). The bottom of a bowl is where it’s flat in every direction, so set both partial slopes (from the maths toolkit) to zero:
\[\frac{\partial\,\text{RSS}}{\partial\beta_0} = -2\sum_i (y_i - \beta_0 - \beta_1x_i) = 0, \qquad \frac{\partial\,\text{RSS}}{\partial\beta_1} = -2\sum_i (y_i - \beta_0 - \beta_1x_i)\,x_i = 0\]
Solving those two equations gives:
\[\hat\beta_1 = \frac{\sum_i (x_i - \bar x)(y_i - \bar y)}{\sum_i (x_i - \bar x)^2}, \qquad \hat\beta_0 = \bar y - \hat\beta_1\bar x\]
\(\bar x\) (“x bar”) is the average of the \(x\) values. The second formula means the best line always passes through the point (average \(x\), average \(y\)).
Slide typo: in equation (15), the bottom must be squared: \(\sum (x_i - \bar x)^{2}\).
Worked example. \(x = (1, 2, 3, 4, 5)\), \(y = (2, 4, 5, 4, 5)\).
- Averages: \(\bar x = 3\), \(\bar y = 4\).
- Subtract them: \(x - \bar x = (-2, -1, 0, 1, 2)\) and \(y - \bar y = (-2, 0, 1, 0, 1)\).
- Top: multiply pairs and add: \(4 + 0 + 0 + 0 + 2 = 6\). Bottom: square and add: \(4 + 1 + 0 + 1 + 4 = 10\).
- \(\hat\beta_1 = 6/10 = 0.6\) and \(\hat\beta_0 = 4 - 0.6 \times 3 = 2.2\).
- Line: \(\hat y = 2.2 + 0.6x\). Predictions \((2.8, 3.4, 4.0, 4.6, 5.2)\).
- Residuals: \((-0.8, 0.6, 1.0, -0.6, -0.2)\). They add to zero (always true with an intercept). RSS \(= 0.64 + 0.36 + 1 + 0.36 + 0.04 = 2.4\).
Technique 3: the matrix formula (many inputs at once)
Problem it solves: with 13 inputs (the Boston data), solving 14 equations by hand is hopeless. Matrices write them all in one line.
Put a 1 at the front of each row of data (so \(\beta_0\) gets multiplied by 1), and stack the rows into the design matrix \(\mathbf X\). Then the predictions are \(\hat{\mathbf y} = \mathbf X\beta\), and setting the gradient to zero gives the normal equations:
\[\mathbf X^T\mathbf X\,\hat\beta = \mathbf X^T\mathbf y \qquad\Rightarrow\qquad \hat\beta = (\mathbf X^T\mathbf X)^{-1}\mathbf X^T\mathbf y\]
The \(^{-1}\) means the inverse of a matrix, the matrix version of “divide by”. It only exists when no input is an exact copy or combination of others, and you have more rows than coefficients.
For a 2×2 matrix you can do it by hand:
\[\begin{pmatrix}a & b\\ c & d\end{pmatrix}^{-1} = \frac{1}{ad - bc}\begin{pmatrix}d & -b\\ -c & a\end{pmatrix}\]
Same data, matrix way. Rows of \(\mathbf X\) are \((1, x_i)\).
\(\mathbf X^T\mathbf X = \begin{pmatrix} n & \sum x\\ \sum x & \sum x^2\end{pmatrix} = \begin{pmatrix}5 & 15\\ 15 & 55\end{pmatrix}\), \(\quad\mathbf X^T\mathbf y = \begin{pmatrix}\sum y\\ \sum xy\end{pmatrix} = \begin{pmatrix}20\\ 66\end{pmatrix}\).
\(ad - bc = 275 - 225 = 50\). So \(\hat\beta = \frac{1}{50}\begin{pmatrix}55 & -15\\ -15 & 5\end{pmatrix}\begin{pmatrix}20\\66\end{pmatrix} = \frac1{50}\begin{pmatrix}110\\30\end{pmatrix} = \begin{pmatrix}2.2\\0.6\end{pmatrix}\). Same answer as before.
Why use several inputs together, not one at a time? The slides’ advertising example: alone, Newspaper spending looks useful (slope 0.055). Together with Radio, it drops to about 0. Newspaper and Radio budgets tend to rise together, so on its own Newspaper was taking credit for Radio’s effect. Each \(\beta_j\) in a multiple regression is the effect with the others held fixed.
Technique 4: gradient descent (walk downhill instead)
Problem it solves: the matrix formula gets slow with huge data, and for many later models (like logistic regression) there is no formula at all. Gradient descent works anyway. It’s what the week 2 lab uses.
The idea: stand somewhere on the bowl, feel which way is uphill (the gradient), take a small step the other way, repeat.
\[\beta^{(t+1)} = \beta^{(t)} - \eta\,\nabla\text{RSS}(\beta^{(t)})\]
- \(\beta^{(t)}\) is where you are after \(t\) steps.
- \(\eta\) (“eta”), the learning rate, is the step size. You choose it.
- For linear regression, \(\nabla\text{RSS} = -2\sum_i (y_i - \hat y_i)\,\tilde x_i\), where \(\tilde x_i\) is row \(i\) with the 1 at the front.
Worked example (one number to find). Use the centred data above, so the model is just \(\hat y = \beta x\) (centring removes the intercept). Then the slope of the RSS is \(-2(6 - 10\beta)\). With \(\eta = 0.01\), starting at \(\beta = 0\):
| step | \(\beta\) | slope of RSS | new \(\beta\) |
|---|---|---|---|
| 0 | 0 | \(-12\) | \(0 + 0.01 \times 12 = 0.12\) |
| 1 | 0.12 | \(-9.6\) | \(0.216\) |
| 2 | 0.216 | \(-7.68\) | \(0.293\) |
| … | heads to \(0.6\) |
Each step closes 20% of the gap to the answer 0.6.
Two things the lab cares about:
- Step size: too big and you overshoot the bottom and fly off (the numbers blow up). Too small and it takes forever. Here it only works if \(\eta < 1/10\).
- Normalise the inputs first. If one input is in thousands and another in fractions, no single step size suits both. That’s why the lab gives you
normalize(). - Starting point doesn’t matter (lab Q1 with 10 random starts): the RSS is one bowl with one bottom, so every start ends in the same place.
Technique 5: judging the fit (RSE and \(R^2\))
Problem it solves: you have a line. Is it any good?
RSE (residual standard error): the typical size of a miss, in the same units as \(Y\).
\[\text{RSE} = \sqrt{\frac{\text{RSS}}{n - p - 1}}\]
Why divide by \(n - p - 1\) and not \(n\)? Because the line was fitted to these same points, its residuals come out a bit too small. Each of the \(p + 1\) numbers you estimated “uses up” one data point’s worth of freedom, so you divide by what’s left.
\(R^2\): the fraction of the variation in \(Y\) that the model explains, from 0 to 1, no units.
\[\text{TSS} = \sum_i (y_i - \bar y)^2, \qquad R^2 = 1 - \frac{\text{RSS}}{\text{TSS}}\]
TSS is how spread out \(Y\) is before you use any inputs (just guessing the average). RSS is what’s left after using the line. In simple regression \(R^2\) equals the correlation squared.
Continuing the example. RSS \(= 2.4\). TSS \(= 4 + 0 + 1 + 0 + 1 = 6\). \(R^2 = 1 - 2.4/6 = 0.6\) (and the week 1 correlation was 0.775, and \(0.775^2 = 0.6\)). RSE \(= \sqrt{2.4/(5 - 1 - 1)} = \sqrt{0.8} \approx 0.89\).
A warning: a high \(R^2\) on training data doesn’t prove the model is good on new data. See the overfitting trap in the big picture.
The estimates wobble. A different sample would give a slightly different \(\hat\beta\). The slides give how much: \(\text{var}(\hat\beta) = \sigma^2(\mathbf X^T\mathbf X)^{-1}\). You don’t need to compute it; just know it gets worse with more noise, less data, or inputs that copy each other. That last case is why ridge regression exists.
Technique 6: dummy variables (using categories as inputs)
Problem it solves: the formula needs numbers, but some inputs are categories (owns a house: yes or no; region: East, West, South).
Two categories: make a column that is 1 for “yes” and 0 for “no”. Then \(\beta_0\) is the average for the “no” group and \(\beta_1\) is the difference between groups.
More than two categories: with \(m\) categories, make \(m - 1\) columns. The category without a column is the baseline, and each \(\beta\) is that category’s difference from the baseline. (Making \(m\) columns breaks the matrix formula, because they always add up to the column of 1s.)
Region with East as baseline: West column, South column. East: \(\beta_0\). West: \(\beta_0 + \beta_1\). South: \(\beta_0 + \beta_2\). If average balances are East 500, West 550, South 620, then \(\beta_0 = 500\), \(\beta_1 = 50\), \(\beta_2 = 120\).
In pandas: pd.get_dummies(df['region'], drop_first=True).
Technique 7: interactions (when one input changes another’s effect)
Problem it solves: plain linear regression assumes each input’s effect is the same regardless of the others. But radio ads might make TV ads work better.
Add a column that is the two inputs multiplied together:
\[Y = \beta_0 + \beta_1X_{TV} + \beta_2X_{Radio} + \beta_3\,X_{TV}X_{Radio} + \epsilon\]
Now one more unit of TV adds \(\beta_1 + \beta_3 X_{Radio}\): more when radio spend is higher. It’s still fitted by ordinary least squares; you just added a column.
Slide typo: equations (48) and (49) leave out the interaction’s own coefficient. It should be \((\beta_{TV} + \beta_3X_{Radio})X_{TV}\).
Technique 8: checking the assumptions with a residual plot
Problem it solves: the model assumes a straight-line shape, effects that simply add up, and luck of roughly the same size everywhere. How do you know if that’s wrong?
Plot residuals (up/down) against predictions (across). Healthy: a shapeless horizontal band. A U shape means the true rule is curved (try polynomial terms). A funnel that widens means the luck gets bigger for bigger values.
What to read in the book
C3C6C7.txt names chapters 3, 6 and 7. For this page, chapter 3:
| Section | Read? | Why |
|---|---|---|
| 3.1.1 Estimating the coefficients | Read | Least squares, the formulas |
| 3.1.2 Accuracy of the coefficient estimates | Skim | Only the idea that \(\hat\beta\) wobbles. Skip standard errors, confidence intervals, t-statistics, p-values |
| 3.1.3 Accuracy of the model | Read | RSE and \(R^2\) |
| 3.2.1 Multiple regression coefficients | Read | Including the newspaper/radio example |
| 3.2.2 Some important questions | Skip | F-statistic and variable selection are not on the slides |
| 3.3.1 Qualitative predictors | Read | Dummy variables |
| 3.3.2 Extensions | Read | Interactions and polynomial terms |
| 3.3.3 Potential problems | Skim | Non-linearity, correlated errors, non-constant variance, collinearity only. Skip outliers, leverage, VIF |
| 3.4 Marketing plan | Skip | |
| 3.5 Linear regression vs KNN | Recommended | For lab Q2, see KNN regression |
Practice prompt
I'm studying CS5100 Data Analysis (textbook ISLP, chapter 3). I want practice questions on LINEAR REGRESSION, done by hand:
- computing the least-squares slope and intercept from 4-6 data points (beta1 = Sxy/Sxx, beta0 = ybar - beta1*xbar)
- residuals, RSS, TSS, R-squared and RSE
- deriving the formulas by setting partial derivatives of RSS to zero
- the matrix form: building the design matrix X, computing X^T X and X^T y, and solving with a 2x2 inverse
- gradient descent steps by hand, choosing a learning rate, and why too large a learning rate diverges; why normalising helps
- interpreting coefficients in multiple regression ("holding other predictors fixed"), and why correlated predictors can change a coefficient
- dummy variables for categories (m-1 dummies, baseline), and interaction terms
- reading a residual plot (U-shape, funnel)
Please:
1. Use plain, everyday language and explain any technical term in one short sentence.
2. Give me ONE question at a time, easy first, then harder. Use small whole numbers so I can work by hand.
3. Wait for my answer, then mark it and show the full working step by step.
4. If I get it wrong, explain what went wrong and give me a similar question.
5. Every 5 questions, summarise my strengths and what to practise next.