KNN regression
Week 1's KNN, but for numbers: instead of a majority vote, take the average answer of the K nearest neighbours.
Question type: regression.
Week: 2 (lab Q2 compares it with linear regression on the Boston housing data).
Book: ISLP 3.5 (recommended; not in the slide headers but it’s exactly lab Q2).
Techniques on this page: neighbourhood average, normalising, the comparison with linear regression.
The problem it solves
Linear regression assumes a straight-line shape. If the true relationship is curvy in ways you can’t guess, that assumption hurts. KNN regression assumes no shape: to predict a house price, it averages the prices of the most similar houses. It’s the non-parametric alternative, and the lab asks you to compare the two.
If you’ve read the KNN classifier page, you already know 90% of this.
Technique 1: the neighbourhood average
- Find the \(K\) training points nearest to the new point \(x_0\) (same distance as week 1).
- Predict the average of their \(y\) values:
\[\hat y(x_0) = \frac1K\sum_{i\,\in\,\text{neighbours}} y_i\]
Worked example. \(x = (1,2,3,4,5,6)\), \(y = (1.0, 1.5, 2.5, 3.0, 5.0, 6.0)\).
- New point \(x_0 = 3.4\), \(K = 3\). Nearest: \(x = 3\) (gap 0.4), \(x = 4\) (0.6), \(x = 2\) (1.4). Prediction \((2.5 + 3.0 + 1.5)/3 \approx 2.33\).
- New point \(x_0 = 5.2\), \(K = 2\). Nearest: \(x = 5\) (0.2), \(x = 6\) (0.8). Prediction \((5.0 + 6.0)/2 = 5.5\).
\(K\) is the same bendiness dial as before. \(K = 1\) has zero training error (every point predicts itself) and means nothing. Large \(K\) flattens everything towards the overall average. Choose \(K\) on a validation set (see ridge regression for the train/validation/test recipe).
Technique 2: normalise first
Distances add up raw differences. In the Boston data, TAX is in the hundreds and NOX is around 0.5, so without rescaling, “nearest” would mean “similar tax” and nothing else. The lab’s normalize() rescales each column to average 0 and spread 1. Strictly, use the training set’s average and spread for all three splits, so no information leaks from the test data.
Technique 3: KNN vs linear regression (lab Q2)
| Situation | Usually wins | Why |
|---|---|---|
| the truth is close to a straight line | linear regression | right shape, low variance |
| strongly curved truth, plenty of data | KNN with a good \(K\) | no wrong shape assumed |
| many inputs (large \(p\)) | linear regression | with many inputs every point is far from every other, so “neighbours” aren’t really near (the curse of dimensionality) |
| you need to explain why | linear regression | coefficients can be read; KNN is a black box |
Compare them fairly: choose each one’s settings on the validation set, then compare on the test set once.
What to read in the book
| Section | Read? | Why |
|---|---|---|
| 3.5 Comparison of linear regression with KNN | Read | Exactly the argument lab Q2 wants, with Figures 3.16 to 3.20 |
| 2.2.3 KNN | Already done | See the KNN page |
Practice prompt
I'm studying CS5100 Data Analysis (textbook ISLP, section 3.5). I want practice questions on KNN REGRESSION:
- predicting by averaging the y values of the K nearest neighbours, from small tables in 1D and 2D
- the effect of K on flexibility, training error and test error
- why features must be normalised, with a numeric example, and why to use the training set's mean and standard deviation
- comparing KNN regression with linear regression: when each wins, the curse of dimensionality, interpretability
- choosing K with a validation set
Please:
1. Use plain, everyday language and explain any technical term in one short sentence.
2. Give me ONE question at a time, easy first, then harder, with small numbers.
3. Wait for my answer, then mark it and show the full working step by step.
4. If I get it wrong, explain the mistake and give me a similar question.
5. Every 5 questions, summarise my strengths and what to practise.