Logistic regression
Take linear regression's straight line and squash it into an S-curve that stays between 0 and 1, so it can be read as a probability. Despite the name, it's for classification.
Question type: classification (the answer is a category).
Week: 3.
Book: ISLP 4.1, 4.2, 4.3 (see the reading table for what to skip).
Techniques on this page: the sigmoid, odds and log-odds, maximum likelihood, the negative log-likelihood, gradient descent, softmax for more than two classes, dummy variables and penalties carried over from week 2.
The problem it solves
You want the chance a customer defaults, given their balance. That’s what the Bayes classifier needs. Why not use linear regression on 0/1 answers?
- A straight line goes on forever. For small balances it predicts negative probabilities; for large ones, above 1. Probabilities have to stay between 0 and 1.
- With more than two categories, the numbers are just labels. Coding eye colour as brown = 0, green = 1, blue = 2 would claim green is “halfway” between the others. Renumber them and linear regression gives different answers. That’s nonsense.
Logistic regression keeps the useful part (a weighted sum of inputs, readable coefficients) and fixes the range problem by bending the line into an S.
Words you’ll hear
| Word | Plain meaning |
|---|---|
| sigmoid, logistic function, \(\sigma(z)\) | the S-shaped squashing function. Any number in, a number between 0 and 1 out |
| \(p(X)\) | the model’s chance that \(Y = 1\) for inputs \(X\) |
| odds | \(p/(1-p)\): how many times more likely yes is than no. \(p = 0.8\) gives odds of 4 |
| log-odds, logit | the log of the odds. In this model it’s a straight line in \(X\) |
| likelihood | how probable the observed data is, according to the model with these coefficients |
| maximum likelihood | choose the coefficients that make the observed data most probable |
| negative log-likelihood (NLL), cross-entropy, log loss | the same thing turned into an error to minimise |
| softmax | the version for more than two categories |
| \(\prod\) | “multiply them all together”, like \(\sum\) means “add them all” |
Technique 1: the sigmoid (squashing a line into a probability)
Problem it solves: keep the prediction between 0 and 1.
Compute a straight-line score \(z = \beta_0 + \beta_1X\) (any number), then squash it:
\[p(X) = \sigma(z) = \frac{1}{1 + e^{-z}}\]
Why it stays in range: \(e^{-z}\) is always positive, so the bottom is always more than 1, so the fraction is always between 0 and 1.
| \(z\) | \(-3\) | \(-2\) | \(-1\) | \(0\) | \(1\) | \(2\) | \(3\) |
|---|---|---|---|---|---|---|---|
| \(\sigma(z)\) | 0.047 | 0.119 | 0.269 | 0.5 | 0.731 | 0.881 | 0.953 |
Patterns to remember: \(\sigma(0) = 0.5\); big positive \(z\) gives nearly 1; big negative gives nearly 0; and it’s symmetric, \(\sigma(-z) = 1 - \sigma(z)\).
Its slope (needed for fitting) has a neat form, found with the chain rule:
\[\sigma'(z) = \sigma(z)\big(1 - \sigma(z)\big)\]
At \(z = 0\) that’s \(0.5 \times 0.5 = 0.25\), the steepest point.
Technique 2: odds and log-odds (reading the coefficients)
Problem it solves: in linear regression, \(\beta_1\) was “the change in \(Y\) per unit of \(X\).” What does it mean now?
With a little algebra, the odds for this model come out as simply \(e^z\). Take logs:
\[\log\frac{p(X)}{1 - p(X)} = \beta_0 + \beta_1X\]
So the model is a straight line, but in log-odds, not in probability. That’s why it counts as a “linear” model.
What \(\beta_1\) means:
- Its sign: positive means the chance rises with \(X\).
- Exactly: increasing \(X\) by 1 multiplies the odds by \(e^{\beta_1}\).
- On the probability itself: not a fixed amount. Near \(p = 0.5\) a change in \(X\) moves \(p\) a lot; near 0 or 1, hardly at all. Precisely, \(dp/dX = p(1-p)\beta_1\).
Decision boundary: with the “predict 1 if \(p \ge 0.5\)” rule, you predict 1 exactly when \(z \ge 0\). So the boundary is where \(\beta_0 + \beta_1x = 0\), i.e. \(x = -\beta_0/\beta_1\). With two inputs it’s a straight line on the chart.
Worked example (slide 24). \(\hat\beta = (-0.5,\ 0.001)\), balance \(x = 1000\).
- Score: \(z = -0.5 + 0.001 \times 1000 = 0.5\).
- Probability: \(\hat p = 1/(1 + e^{-0.5}) = 1/1.6065 = 0.6225\). Predict default.
- Odds: \(e^{0.5} = 1.649\) (and \(0.6225/0.3775 = 1.649\), it checks).
- Another £100 multiplies the odds by \(e^{0.1} = 1.105\), a 10.5% rise.
- Boundary: \(x = 0.5/0.001 = 500\). Balances over £500 are predicted to default.
Technique 3: maximum likelihood (what “best coefficients” means now)
Problem it solves: least squares made sense for numbers. For yes/no answers with probability predictions, a better question is: which coefficients make what actually happened look most likely?
For one customer with predicted chance \(p_i\): if they defaulted, the model gave that outcome chance \(p_i\); if not, \(1 - p_i\). A compact way to write both cases is \(p_i^{\,y_i}(1-p_i)^{1-y_i}\) (try \(y_i = 1\) and \(y_i = 0\): anything to the power 0 is 1). Multiply over all customers to get the likelihood:
\[\ell(\beta) = \prod_{i=1}^n p_i^{\,y_i}(1 - p_i)^{1 - y_i}\]
Three customers, actual \(y = (1, 0, 1)\).
- Model A predicts \(p = (0.9, 0.2, 0.8)\): \(\ell = 0.9 \times 0.8 \times 0.8 = 0.576\).
- Model B predicts \(p = (0.6, 0.5, 0.5)\): \(\ell = 0.6 \times 0.5 \times 0.5 = 0.15\).
A makes the real outcomes nearly four times more likely, so maximum likelihood prefers A.
A general principle from slide 20: least squares is itself maximum likelihood when the luck \(\epsilon\) is bell-shaped. So this isn’t a new idea replacing week 2; week 2 was a special case.
Technique 4: the negative log-likelihood (turning it into an error)
Problem it solves: multiplying thousands of small numbers underflows to zero on a computer, and products are awkward to differentiate.
Take the log (products become sums, and the best \(\beta\) doesn’t change because log only ever goes up). Flip the sign so that smaller is better, like the RSS:
\[L(\beta) = -\sum_{i=1}^n\Big[y_i\log p_i + (1 - y_i)\log(1 - p_i)\Big]\]
Each customer contributes \(-\log(\text{chance the model gave to what really happened})\). Confident and right: close to 0. Confident and wrong: big (\(-\log 0.01 = 4.6\)).
Model A above: \(L = -\log 0.576 = 0.55\). Model B: \(L = -\log 0.15 = 1.90\).
Notation clash: on the slides \(\ell\) is the likelihood and you minimise \(-\log\ell\). In the lab, \(\ell\) already means the negative log-likelihood. Same thing, different letter.
Technique 5: gradient descent (because there’s no formula)
Problem it solves: for linear regression we set the slope to zero and solved. Here the coefficients are trapped inside the sigmoid, so the equations can’t be solved by algebra. We walk downhill instead, exactly as in week 2.
Using the chain rule three times and cancelling, the gradient simplifies beautifully:
\[\nabla L(\beta) = -\sum_{i=1}^n (y_i - p_i)\,\tilde x_i\]
where \(\tilde x_i = (1, x_i)\). That’s “actual minus predicted, times the inputs”: the same shape as week 2’s gradient. Then step: \(\beta \leftarrow \beta - \eta\,\nabla L\).
One step by hand. Data \(x = (-1, 0, 1, 2)\), \(y = (0, 0, 1, 1)\). Start \(\beta = (0, 0)\), step size \(\eta = 0.1\).
- All scores are 0, so every \(p_i = 0.5\). \(y - p = (-0.5, -0.5, 0.5, 0.5)\).
- Intercept part: \(-\sum(y_i - p_i) = 0\).
- Slope part: \(-\sum(y_i - p_i)x_i = -(0.5 + 0 + 0.5 + 1) = -2\).
- New \(\beta = (0, 0) - 0.1 \times (0, -2) = (0, 0.2)\).
The loss drops from 2.77 to 2.40. Keep going and it keeps falling.
Lab bug: grad_log_likelihood uses np.sum(...) with no axis, which adds every number into one, so all coefficients get the same update. It should be np.sum(..., axis=0).
Technique 6: softmax (more than two categories)
Problem it solves: the sigmoid handles yes/no. For several categories, give each its own score \(z_k\) and turn the scores into chances that add to 1:
\[\hat p_k = \frac{e^{z_k}}{\sum_{k'} e^{z_{k'}}}\]
Each \(e^{z_k}\) is positive; dividing by the total makes them add to 1. With two categories this is exactly the sigmoid.
Scores \(z = (2, 1, 0)\). \(e^2 = 7.389\), \(e^1 = 2.718\), \(e^0 = 1\), total \(11.107\). Chances \((0.665, 0.245, 0.090)\). Predict the first category.
Technique 7: tools carried over from week 2
- Several inputs: \(p(x) = \sigma(\beta_0 + \beta_1x_1 + \dots + \beta_px_p)\). Each \(\beta_j\) is the change in log-odds per unit of \(x_j\), others held fixed.
- Dummy variables for categories like
student(lab Q1). - An \(L_2\) penalty, exactly like ridge: add \(\lambda\sum_{j\ge1}\beta_j^2\), whose gradient adds \(2\lambda\beta_j\) (lab Q2). Don’t penalise the intercept.
The student paradox (read before lab Q1): on its own, being a student raises the default chance. Add balance, and being a student lowers it. Both are true: students tend to carry bigger balances, and balance drives defaults. Same lesson as week 2’s newspaper and radio.
How it compares
| Logistic regression | LDA | Naive Bayes | KNN | |
|---|---|---|---|---|
| estimates the chances by | S-curve of a straight line | bell curves per class, then Bayes’ theorem | one-input-at-a-time pictures per class | counting neighbours |
| boundary shape | straight | straight | can curve | any shape |
What to read in the book
C4.txt names chapter 4.
| Section | Read? | Why |
|---|---|---|
| 4.1 Overview of classification | Read | Introduces the Default data the lab uses |
| 4.2 Why not linear regression? | Read | The “why” above |
| 4.3.1 The logistic model | Read | Sigmoid, odds, log-odds |
| 4.3.2 Estimating the coefficients | First half | The likelihood. Skip standard errors, z-statistics, p-values |
| 4.3.3 Making predictions | Read | Includes the student dummy |
| 4.3.4 Multiple logistic regression | Read | The student paradox |
| 4.3.5 Multinomial logistic regression | Read the softmax part | The “baseline class” version is a skim |
| 4.6 Generalised linear models | Skip | Not on the slides |
Practice prompt
I'm studying CS5100 Data Analysis (textbook ISLP, section 4.3). I want practice questions on LOGISTIC REGRESSION, done by hand:
- why linear regression fails for classification
- computing the sigmoid sigma(z) = 1/(1+e^-z) for given coefficients and inputs, and the predicted class
- odds and log-odds: converting between p, odds and log-odds; e^beta as the odds multiplier; dp/dx = p(1-p)beta
- finding the decision boundary x = -beta0/beta1, and the x that gives a chosen probability
- deriving sigma'(z) = sigma(z)(1 - sigma(z)) with the chain rule
- writing and computing the likelihood and negative log-likelihood for 2-4 data points; comparing two models by likelihood
- the gradient -sum (y_i - p_i) x_i and doing one or two gradient descent steps by hand
- softmax for 3 classes, and why softmax with 2 classes is the sigmoid
- adding an L2 penalty to the gradient (intercept not penalised)
- the student paradox / why a coefficient can change sign when another predictor is added
Please:
1. Use plain, everyday language and explain any technical term in one short sentence.
2. Give me ONE question at a time, easy first, then harder. Use friendly numbers and tell me the e^x values I'll need if they aren't obvious.
3. Wait for my answer, then mark it and show the full working step by step.
4. If I get it wrong, explain the mistake and give me a similar question.
5. Every 5 questions, summarise my strengths and what to practise.