The big picture
Every model in this course is an attempt to learn an unknown rule from noisy examples. This stage gives you the frame, and the vocabulary the exam expects you to use.
By the end of this stage you can say what statistical learning is, write the basic model \(Y = f(X) + \epsilon\), and sort any problem into supervised or unsupervised, regression or classification, prediction or inference, with a concrete example each time. That is exactly what the “explain X with a real-world example” exam questions want.
Book: what to read for this stage
ISLP chapter 1 (the Wage, Smarket and gene-expression examples, and the Notation section), and 2.1.1, 2.1.2, 2.1.4, 2.1.5. Skip: the history, “This Book”, “Who should read”, data sets and website pages of chapter 1, and the semi-supervised paragraph in 2.1.4. Skim 2.1.3 (accuracy vs interpretability): one idea, flexible models are harder to interpret.
1. Learning a rule from examples
Statistical learning (machine learning) is a set of tools for understanding data. We have inputs \(X = (X_1, \dots, X_p)\) and an output \(Y\), and we assume
\[Y = f(X) + \epsilon.\]
- \(f\) is a fixed but unknown function: the systematic part, the rule we want.
- \(\epsilon\) is random error with mean zero, independent of \(X\): the luck we can never predict.
Everything we fit is an estimate \(\hat f\) of \(f\), built from training data \(D = \{(x_i, y_i)\}_{i=1}^{n}\).
Words you’ll hear (all mean the same thing)
| Inputs \(X\) | Output \(Y\) |
|---|---|
| predictors, features, independent variables, variables | response, dependent variable, label, target |
\(n\) = number of data points (rows), \(p\) = number of inputs (columns), \(x_{ij}\) = input \(j\) of data point \(i\).
Exam notation
The past papers write the data as \(D = \{(X_n, Y_n)\}_{n=1}^{N}\) and the model’s parameters as \(\lambda\) (for example \(f(X) = \lambda_0 + \lambda_1 X\)). The book and slides write \(i = 1, \dots, n\) and \(\beta\). They mean the same thing; in the exam, use the letters the question uses.
Why can’t a perfect model be perfect? Even if \(\hat f = f\) exactly, a new \(Y\) still differs from \(f(X)\) by \(\epsilon\). ISLP calls this the irreducible error, \(\mathrm{Var}(\epsilon)\). The part of the error caused by \(\hat f \neq f\) is the reducible error, which better models can shrink.
A model predicts house prices. Even after you find the true \(f\), its predictions are still sometimes wrong. Why?
With the true \(f\) the reducible error is zero by definition. What’s left is \(\epsilon\), which no model built from \(X\) can predict. Small training sets and overfitting are reasons \(\hat f\) differs from \(f\), which is the reducible part.
2. Prediction or inference?
We estimate \(f\) for one of two reasons:
| Prediction | Inference | |
|---|---|---|
| The question | “What will \(Y\) be for this new \(X\)?” | “How does \(Y\) depend on \(X\)?” |
| Does the form of \(\hat f\) matter? | No: a black box is fine if it’s accurate | Yes: \(\hat f\) must be interpretable |
| Example | From a blood sample \(X\), predict the risk \(Y\) of a bad drug reaction | From TV, radio and newspaper spend \(X\), find which media actually raise sales \(Y\), and by how much |
Typical inference questions: which predictors matter? Is the relationship with each one positive or negative? Is it linear?
Worked example: sorting real problems.
- A bank wants to know whether income or age matters more for loan defaults. “Which matters more” is about how \(Y\) depends on \(X\): inference.
- A shop wants tomorrow’s sales figure for each store. Only the number matters: prediction.
- A hospital wants to flag high-risk patients, and doctors want to see why each was flagged. Both: accurate predictions, but the doctors need an interpretable model, so this pushes towards inference-friendly models.
A charity has data on 2,000 past donors (age, income, region, number of past gifts) and wants to understand which of these factors are associated with giving larger amounts. Which description fits best?
The output (amount given) is a number, so regression. “Understand which factors are associated” is about how \(Y\) depends on \(X\), so inference. There is an output, so it’s supervised.
3. Parametric or non-parametric?
Parametric methods work in two steps:
- Assume a form for \(f\), for example linear: \(f(X) = \beta_0 + \beta_1 X_1 + \dots + \beta_p X_p\).
- Fit it: choose the few numbers \(\beta_j\) so that \(\hat f\) matches the training data (for example, by least squares in week 2).
Good: estimating \(f\) shrinks to estimating a few numbers, which needs less data and is easy to interpret. Bad: if the assumed form is wrong (the truth is curved, the model is a line), \(\hat f\) is systematically off, whatever the data.
Non-parametric methods (KNN, trees, smoothing splines) assume no particular form. \(\hat f\) follows the data as closely as possible without being too wiggly. Good: no risk of assuming the wrong shape. Bad: they need many more observations, and you must choose how smooth \(\hat f\) is, which isn’t easy. Too rough and it follows the noise: that’s overfitting.
Why ever choose a restrictive model? (The slides list three reasons.) It’s interpretable, it’s cheap to compute, and it’s less likely to overfit.
Tick every statement that is true.
Non-parametric methods need more data, because they have to learn the shape as well as the values. Flexible models are usually harder to interpret (ISLP 2.1.3). The other three are true.
4. Supervised or unsupervised? Regression or classification?
Supervised: every training input \(x_i\) comes with its output \(y_i\), and we learn the input-to-output relationship. Unsupervised: we only have the inputs; there’s no output to “supervise” the fit. We look for structure, especially clustering (do the points fall into groups?).
The three motivating examples from the slides:
| Data set | Goal | Output | Type |
|---|---|---|---|
| Wage | predict a worker’s wage from age, year, education | a number | supervised, regression |
| Smarket | will the stock index go up or down today, from the last few days’ changes? | up / down | supervised, classification |
| NCI60 gene expression | find groups among 64 cancer cell lines from 6,830 measurements | none | unsupervised (clustering) |
Regression vs classification is decided by the output: a quantitative output (age, price, income) means regression; a qualitative or categorical output (eye colour, default yes/no, a digit) means classification. The type of the inputs matters much less, because categories can be encoded as numbers.
Some methods only do one job (linear regression: regression; logistic regression, despite the name: classification). Others do both: KNN and decision trees.
Worked example: an exam-style answer. “What is the difference between supervised and unsupervised learning? Give a real-world example of each.” (2023 Q1a)
In supervised learning each training example comes with the correct output, and we learn a rule from inputs to output; for example, predicting a house’s sale price (output, a number) from its floor area and number of rooms (inputs), using past sales. In unsupervised learning there is no output, only inputs, and we look for structure such as groups; for example, a supermarket grouping its customers into segments from their shopping histories, with no labels saying which segment anyone is in.
Notice the shape: a one-line definition of each, then a specific example with the inputs and output named. That’s what earns the marks.
A streaming service records each user’s viewing history and wants to discover groups of users with similar taste, without any predefined categories. This is:
No output is recorded for any user: we are looking for groups in the inputs alone. The groups aren’t known in advance, so there’s nothing to supervise the fit.
Predicting a patient’s blood pressure (in mmHg) from their age, weight and smoking status (yes/no) is:
The output decides: blood pressure is a number, so it’s regression. A yes/no input is simply encoded as 0/1.
Stage test
(5 marks, exam format 1.) Explain the difference between supervised and unsupervised learning, and give a real-world example of each.
In supervised learning every training example \((x_i, y_i)\) includes the correct output, and the goal is to learn a function \(f\) with \(Y \approx f(X)\), either to predict \(Y\) for new inputs or to understand how \(Y\) depends on \(X\). Example: predicting a used car’s price from its age and mileage, trained on past sales. In unsupervised learning there is no output: we only observe the inputs and look for structure, such as groups of similar observations (clustering). Example: grouping shoppers into segments using their purchase histories, where no segment labels exist.
In \(Y = f(X) + \epsilon\), which statement is correct?
\(\epsilon\) is the irreducible noise: mean zero, independent of \(X\), unpredictable from \(X\). \(f\) is fixed but unknown. \(\hat f - f\) is the reducible error.
A company has 20 past product launches, each recorded as success or failure with the price, marketing budget, competitor price and 10 other variables. It wants to know whether a new product will succeed. What are the problem type and \((n, p)\)?
Success/failure is categorical: classification. “Will it succeed” is prediction. 20 launches is \(n\); the inputs are price, budget, competitor price plus 10 others, so \(p = 13\) (success/failure is the output, not an input). This is ISLP 2.4 exercise 2(b).
Which is the best reason to prefer a restrictive (e.g. linear) model over a very flexible one when your goal is inference?
For inference you need to read the relationship off the model, which a linear model makes easy. It does not always have lower test error: if the truth is very non-linear, a flexible model may predict better.
Tick every method that can be used for both regression and classification.
KNN averages neighbours (regression) or takes a vote (classification). Trees predict a leaf’s mean or majority class. Linear regression predicts numbers; logistic regression predicts class probabilities, so it’s a classifier.
(3 marks.) Explain the difference between a parametric and a non-parametric method, with one advantage of each.
A parametric method assumes a form for \(f\), such as \(f(X) = \beta_0 + \beta_1 X_1 + \dots + \beta_p X_p\), and then estimates its handful of parameters from the data; this needs relatively little data and is easy to interpret, but it is biased if the assumed form is wrong. A non-parametric method (e.g. KNN, decision trees) makes no assumption about the shape of \(f\) and lets the data decide it, so it can fit complicated relationships, but it needs many more observations and a choice of how smooth \(\hat f\) should be, otherwise it overfits.