Start here: the big picture
Not a model. This is the frame that every model in the course fits inside. Read it once and the rest of the course stops appearing out of thin air.
Why this page exists
Week 1’s slides jump between examples, notation, calculus and the Bayes classifier. They look unrelated, but they’re all pieces of one story: we want to guess an answer from some information, and we want to know how good our guess is. Every model after this is just a different way of making the guess.
1. The one idea behind the whole course
You have some information about something (a person’s age and years of education) and you want to guess an answer (their income). The course writes this as:
\[Y = f(X) + \epsilon\]
Read it out loud as: “the answer equals some rule applied to the information, plus some luck.”
- \(X\) is the information you have (the inputs). It can be several things, \(X_1, X_2, \dots, X_p\).
- \(Y\) is the answer you want (the output).
- \(f\) is the true rule connecting them. Nobody knows it. Our whole job is to estimate it.
- \(\epsilon\) (the Greek letter “epsilon”) is the luck: things that affect \(Y\) that \(X\) doesn’t capture. Two people with the same education still earn different amounts. It averages out to zero.
Our estimate of the rule is written \(\hat f\) (“f-hat”). The hat always means “our estimate”. So \(\hat y\) is a predicted answer and \(\hat\beta\) is an estimated number.
Words you’ll hear, in plain English
| Word | What it means |
|---|---|
| predictor, feature, variable, input | a piece of information, one column of the spreadsheet |
| response, output, target | the thing you’re trying to guess |
| observation, data point | one row of the spreadsheet (one person, one car) |
| \(n\) | how many rows (people) you have |
| \(p\) | how many columns of information (predictors) you have |
| \(x_{ij}\) | the value in row \(i\), column \(j\) |
| training data | the rows you learn from |
| test data | new rows the model has never seen, used to check it honestly |
| model | a shape you assume the rule \(f\) has, with some numbers left to fill in |
| parameters, coefficients | the numbers you fill in, often called \(\beta\) (“beta”) |
| fitting, training | choosing those numbers using the training data |
2. Two kinds of question
Every problem in this course is one of these. Look at what kind of thing the answer is.
| If the answer is… | It’s called | Examples | Models for it |
|---|---|---|---|
| a number | regression | income, house price, wage | linear regression, polynomial regression, ridge, KNN regression |
| a category | classification | default yes/no, spam or not, eye colour | Bayes classifier, KNN, logistic regression, LDA, naive Bayes |
(Logistic regression is a classification model despite its name. That name confuses everyone.)
There’s also a third kind where there is no answer column at all, and you just look for groups in the data. That’s unsupervised learning (e.g. clustering). Weeks 1 to 3 only touch it in one example; everything else is supervised, meaning you have the answers for your training rows.
3. Two reasons to build a model
- Prediction: you only care that the guess is accurate. The model can be a black box. (“Will this patient react badly to the drug?”)
- Inference (understanding): you want to know how the inputs affect the answer. The model must be readable. (“Which kind of advert actually increases sales, and by how much?”)
This matters later: simple models like linear regression are easy to read, so they’re good for understanding. Flexible models like KNN can predict well but tell you little about why.
4. Two ways to estimate the rule
Parametric: assume a shape, then fill in a few numbers.
“I’ll assume income is a straight line in education: \(f(X) = \beta_0 + \beta_1 X\).” Now you only need two numbers. Easy, needs less data, easy to read. The risk: if the true rule is curved, your straight line is wrong no matter how much data you have.
Non-parametric: don’t assume a shape, just follow the data.
KNN is the example: “to guess for a new person, look at the most similar people I already know.” No shape assumed, so it can follow any pattern. The cost: it needs a lot more data, and you have to decide how smooth or bumpy to let it be.
5. How we score a model
You can’t improve what you can’t measure. There are two scores, one for each kind of question.
Technique: mean squared error (MSE), for regression
Problem it solves: “My guesses are off by different amounts for different people. Give me one number for how wrong I am overall.”
Take each error (guess minus truth), square it so negatives don’t cancel positives and big misses count extra, then average:
\[\text{MSE} = \frac1n \sum_{i=1}^n \big(\hat y_i - y_i\big)^2\]
The \(\sum\) (“sigma”) just means add up over every row \(i\) from 1 to \(n\).
Worked example. Truth \(y = (3, 5, 2, 8)\), guesses \(\hat y = (2.5, 5, 3, 7)\). Errors: \(-0.5,\ 0,\ 1,\ -1\). Squared: \(0.25,\ 0,\ 1,\ 1\). Total \(2.25\). Divide by 4: MSE = 0.5625.
Technique: error rate, for classification
Problem it solves: the same, but when answers are categories, “how far off” makes no sense. You’re either right or wrong.
\[\text{error rate} = \frac1n\sum_{i=1}^n \mathbb{1}(\hat y_i \neq y_i)\]
The \(\mathbb{1}(\dots)\) is the indicator: it’s 1 if the thing in brackets is true and 0 if not. So the sum just counts the mistakes, and dividing by \(n\) makes it a fraction. Accuracy is \(1 - \text{error rate}\).
Worked example. Truth (A, B, B, A, C), guesses (A, B, A, A, C). One wrong out of five, so error rate \(= 0.2\) and accuracy \(= 0.8\).
6. The trap: training score vs test score
This is the single most important idea in the course, and it explains why so many later techniques exist.
A model that is allowed to be very bendy can memorise the training data, including the luck \(\epsilon\). Its training score looks perfect, but on new data it does badly, because the luck it memorised doesn’t repeat. This is overfitting.
So the score that matters is on test data (rows the model never saw). The training score is a misleading guide.
Why later weeks exist
Almost every “new technique” from week 2 onward is an answer to this trap:
- Validation sets (week 2): hold some data back to choose settings honestly.
- Ridge regression (week 2): deliberately stop a model from bending too much.
- Choosing \(K\) in KNN (weeks 1 to 3): \(K\) is literally a bendiness dial.
7. Bias and variance: the two ways to be wrong
The slides state this without much explanation:
\[\text{test MSE} = \text{bias}^2 + \text{variance} + \text{irreducible error}\]
In plain words, a model can be wrong for three reasons:
- Bias: it’s too stiff. It assumed the wrong shape (a straight line for a curve). Even with infinite data it would be off. Simple models have high bias.
- Variance: it’s too jumpy. Give it a slightly different training set and it changes a lot, because it was chasing the luck. Bendy models have high variance.
- Irreducible error: the luck itself. No model can predict \(\epsilon\). This is a floor nobody can beat.
Making a model more flexible lowers bias but raises variance. So if you plot test error against flexibility you get a U shape: too stiff on the left, too jumpy on the right, best in the middle. You’ll see this U shape in KNN, polynomial regression and ridge regression. It’s the same idea every time.
8. Where each model sits
| Model | Week | Question type | Shape assumed? | Page |
|---|---|---|---|---|
| Bayes classifier | 1 | classification | the “perfect” ideal, not usable directly | Bayes classifier |
| KNN | 1, 2, 3 | both | no | KNN, KNN regression |
| Linear regression | 2 | regression | yes, a straight line or flat plane | Linear regression |
| Polynomial and step regression | 2 | regression | yes, curves built from pieces | Polynomial and step regression |
| Ridge regression | 2 | regression | yes, plus a “don’t bend too much” rule | Ridge regression |
| Logistic regression | 3 | classification | yes, an S-curve of probability | Logistic regression |
| LDA | 3 | classification | yes, bell curves per class | LDA |
| Naive Bayes | 3 | classification | yes, inputs independent per class | Naive Bayes |
9. What to read in the book (ISLP), and what to skip
The file C1C2.txt names chapters 1 and 2. Not all of those chapters are on the slides:
| Section | Read? | Why |
|---|---|---|
| Ch. 1: Wage, Stock Market, Gene Expression examples | Read | Same examples as the slides |
| Ch. 1: Notation and Simple Matrix Algebra | Read | The course uses this notation exactly |
| Ch. 1: history, “This Book”, “Who should read”, data sets, website | Skip | Not on the slides |
| 2.1.1 Why estimate \(f\)? | Read | Prediction vs inference, reducible vs irreducible error |
| 2.1.2 How do we estimate \(f\)? | Read | Parametric vs non-parametric |
| 2.1.3 Accuracy vs interpretability | Skim | One slide on flexibility covers it |
| 2.1.4 Supervised vs unsupervised | Read | Skip the semi-supervised paragraph |
| 2.1.5 Regression vs classification | Read | |
| 2.2.1 Measuring quality of fit | Skim | MSE, training vs test error |
| 2.2.2 Bias-variance trade-off | Skim | Slides only state the formula. Know the meaning, skip the derivation |
| 2.2.3 The classification setting | Read | Bayes classifier and KNN, see their pages |
| 2.3 Lab | Optional | The course lab covers the same Python |
10. Practice prompt
Paste this into any AI chat when you want practice questions.
I'm studying CS5100 Data Analysis (textbook: "An Introduction to Statistical Learning with Python", ISLP, chapters 1-2). I want practice questions on the BIG PICTURE ideas, not a specific model:
- the setup Y = f(X) + epsilon, and what f-hat means
- regression vs classification (deciding from the type of the answer)
- prediction vs inference
- parametric vs non-parametric
- computing MSE and error rate by hand from small lists of numbers
- training error vs test error, and overfitting
- bias, variance and irreducible error, and the U-shaped test error curve
Please:
1. Use plain, everyday language. If you use a technical term, explain it in one short sentence.
2. Give me ONE question at a time, starting easy and getting harder. Mix short "which type of problem is this?" scenarios with small calculations.
3. Wait for my answer. Then tell me if I'm right, and show the full worked solution step by step.
4. If I get one wrong, give me a similar question before moving on.
5. After every 5 questions, tell me which ideas I seem solid on and which need more work.