Start here

CS5100 Model Map

One page per model, showing what it's for, which techniques live inside it, and how it connects to the others. Start here when the lectures feel like they're pulling ideas out of thin air.

The pattern behind the course

You spotted it already: the course teaches models, and inside each model, techniques.

A model is a shape you assume for the rule connecting inputs to the answer. “It’s a straight line” (linear regression). “It’s an S-curve of probability” (logistic regression). “Copy your nearest neighbours” (KNN).
A technique is a tool for using that model: how to find its numbers (least squares, gradient descent, maximum likelihood), how to stop it overreacting (ridge penalty, choosing \(K\)), or how to score it (MSE, ROC).

The same handful of techniques keeps reappearing across models. Once you know them, each “new” model is mostly old pieces in a new arrangement.

🧭New: the guided learning pathWeeks 1 to 4 in 18 stages, with lessons, worked examples and exam-style quizzes. Each stage unlocks when you pass the one before, ending in a mock exam.

Pick a page

Start here: the big picture

The frame every model fits in: the rule, the luck, scoring, overfitting, bias vs variance.

Maths toolkit

Vectors, slopes, the chain rule and gradients, and why a data course needs them.

Bayes classifier

The ideal: always pick the most likely category. Every classifier tries to copy it.

KNN (classifier)

Ask your K nearest neighbours and go with the majority. Also used again in week 3.

Linear regression

The closest straight line. Least squares, the matrix formula, gradient descent, R².

Polynomial & step regression

Bend the line by inventing new columns: x², x³, or "is x in this range?".

Ridge regression

Linear regression with a "keep coefficients small" rule, so it stops chasing noise.

KNN regression

KNN for numbers: average the neighbours. Compared with linear regression in lab Q2.

Logistic regression

Squash a line into an S-curve of probability. Maximum likelihood and gradient descent.

LDA

Describe each class with a bell curve, then flip it with Bayes’ theorem.

Naive Bayes

LDA’s shortcut cousin: treat each input separately and multiply.

Judging classifiers

Confusion matrix, thresholds, ROC curves and AUC. Used in the week 3 lab.

Regression trees

Yes/no questions cut the inputs into boxes; predict each box’s average. Greedy splitting by RSS.

Classification trees

Boxes that vote. Gini, entropy and the error rate, and why Gini grows better trees.

Tree pruning

Grow too big, then cut back with a fee per leaf (α). Weakest-link pruning and sklearn’s ccp_alpha.

Neural network

Invent features with bent weighted sums (ReLU, sigmoid), then fit a line on them.

Neural network classifier

One output per class, softmax and cross-entropy. Plus the lab’s double-softmax bug.

Training neural networks

Random starts, gradient descent, backprop, early stopping, and the PyTorch training loop.

Which techniques belong to which model

Model	How its numbers are found	How you control bendiness	How you score it
Bayes classifier	nothing to fit: needs the true chances (never known)	n/a	Bayes error rate (the floor)
KNN	nothing to fit: just stores the data	\(K\), chosen on a validation set	error rate
Linear regression	least squares, by formula or gradient descent	add or remove inputs	RSS, RSE, \(R^2\), test MSE
Polynomial and step	least squares on new columns	degree or number of cut points	test MSE
Ridge regression	least squares plus a penalty	\(\lambda\), chosen on a validation set	test MSE
KNN regression	nothing to fit	\(K\)	test MSE
Logistic regression	maximum likelihood, by gradient descent	inputs, optional penalty \(\lambda\)	error rate, ROC, AUC
LDA	counting and averaging (priors, means, pooled spread)	(fixed by its assumption)	error rate, ROC, AUC
Naive Bayes	counting and averaging per input	(fixed by its assumption)	error rate, ROC, AUC
Regression tree	greedy splits that cut the RSS most; leaf averages	depth, leaf size, pruning fee \(\alpha\)	test MSE, \(R^2\)
Classification tree	greedy splits that cut the Gini index most; leaf votes	depth, pruning fee \(\alpha\)	error rate
Neural network	gradient descent + backpropagation	hidden units \(K\), penalty \(\lambda\), early stopping	test MSE
Neural network classifier	gradient descent on cross-entropy	same as above	error rate, confusion matrix

Techniques that keep coming back

Technique	What problem it solves	Appears in
Setting the slope to zero	finds the bottom of an error “valley” exactly	maths toolkit, linear regression, ridge
Gradient descent	walks downhill when there’s no formula, or the data is huge	linear regression (lab), logistic regression (lab), neural networks
Dummy variables	lets categories be used as inputs	linear regression, logistic regression (lab Q1), step functions
Normalising inputs	puts inputs on the same scale	KNN, gradient descent, ridge, neural networks (not needed for trees)
Train / validation / test split	chooses settings honestly	KNN’s \(K\), ridge’s \(\lambda\), polynomial degree, pruning’s \(\alpha\), a network’s \(K\) and stopping epoch
Penalty for complexity	trades training fit for simplicity	ridge (\(\lambda\sum\beta^2\)), tree pruning (\(\alpha\lvert T\rvert\)), network weight decay
The chain rule	slopes of functions inside functions	logistic regression’s gradient, backpropagation
Softmax + cross-entropy	probabilities for many classes, and how to train them	multinomial logistic regression, neural network classifier
Bias vs variance (the U shape)	explains why “more flexible” isn’t always better	every model
Bayes classifier rule (pick the most likely)	turns chances into a decision	KNN, logistic regression, LDA, naive Bayes
Bayes’ theorem	flips “inputs given class” into “class given inputs”	LDA, naive Bayes

How the weeks build on each other

Week 1 sets the stage. What a model is, how to score it, why the test score matters more than the training score, and the ideal classifier we’re all trying to copy. KNN is the first real model: it copies the ideal by counting neighbours.
Week 2 builds the first model with numbers to fit, the straight line. Then it bends it (polynomials, steps) and calms it down when it bends too much (ridge).
Week 3 goes back to categories. Three new ways to estimate the chance of each class: squash a line into an S-curve (logistic), or describe each class and flip with Bayes’ theorem (LDA, naive Bayes). Then how to judge them properly.
Week 4 brings two flexible models that work for numbers and categories. Trees cut the inputs into boxes with yes/no questions (and get pruned with a penalty, like ridge). Neural networks learn their own features and are fitted with gradient descent, using the chain rule (backpropagation) and week 3’s softmax.

How to use these pages

Read the green “why this exists” box first on every page. That’s the context the lectures skip.
Purple boxes are plain-English word lists. Yellow boxes are worked examples to copy out by hand. Red boxes flag slide typos and lab bugs.
Turn on Try-first (top bar) to hide every worked example until you’ve had a go on paper.
Press Mark as understood on each technique. Your progress shows in the sidebar and on these tiles (it’s saved in this browser only).
Play with the interactive demos: sliders that move the S-curve, roll gradient descent downhill, shift a decision boundary and more.
Each page ends with a practice prompt. Click “Copy prompt” and paste it into any AI chat to get questions one at a time, with worked solutions after you answer.
Each page also says which parts of the book chapter are on the slides and which you can skip.

Built from the week 1 to 4 slides, lab notebooks and chapter lists (C1C2, C3C6C7, C4; week 4’s chapters 8.1 and 10 come from the slide headers). Pages for later weeks can be added in new w5, w6… folders.