Fitting logistic regression
Choose the parameters that make what actually happened as probable as possible. That's maximum likelihood; taking logs turns it into the negative log-likelihood, which gradient descent minimises. Then softmax handles more than two classes.
2023 Q3 asked for the objective used to train logistic regression; the 2025 neural network question asked for an objective function too. The negative log-likelihood (cross-entropy) is the answer to both whenever the output is a class.
Book: what to read for this stage
ISLP 4.3.2 estimating the coefficients: read the first half (the likelihood, eq. 4.5). Skip Table 4.1’s standard errors, z-statistics and p-values. 4.3.5 multinomial logistic regression: read the softmax version (eq. 4.13); skim the baseline-class version (eq. 4.10).
Slide and lab notes
On the slides \(\ell\) is the likelihood and \(-\log\ell\) is minimised; in the lab notebook \(\ell\) already means the negative log-likelihood. Lab bug: grad_log_likelihood ends with np.sum(...) without axis=0, so it returns one number instead of a length-3 vector, and every coefficient gets the same update. Fix: np.sum(..., axis=0).
1. The likelihood
For one training point with model probability \(p_i = \sigma(\tilde x_i^T\beta)\), the probability the model gave to what actually happened is \(p_i\) if \(y_i = 1\) and \(1 - p_i\) if \(y_i = 0\). Both at once: \(p_i^{y_i}(1 - p_i)^{1 - y_i}\). Treating points as independent, the probability of the whole data set is the likelihood:
\[\ell(\beta) = \prod_{i=1}^{n}p_i^{\,y_i}(1 - p_i)^{1 - y_i}.\]
Maximum likelihood chooses the \(\beta\) that makes \(\ell\) biggest.
Worked example. \(y = (1, 0, 1)\). Model A gives \(p = (0.9, 0.2, 0.8)\): \(\ell = 0.9 \times 0.8 \times 0.8 = 0.576\). Model B gives \(p = (0.6, 0.5, 0.5)\): \(\ell = 0.6 \times 0.5 \times 0.5 = 0.15\). A explains the data almost four times better.
\(y = (1, 0)\) and the model gives \(p = (0.8, 0.3)\). Compute the likelihood.
\(0.8 \times (1 - 0.3) = 0.8 \times 0.7 = 0.56\).
2. The negative log-likelihood
Products of many numbers below 1 underflow to 0 on a computer and are awkward to differentiate. Take logs (products become sums; the best \(\beta\) doesn’t change, since log is increasing) and flip the sign so we minimise:
\[\boxed{L(\beta) = -\log\ell(\beta) = -\sum_{i=1}^{n}\Big[y_i\log p_i + (1 - y_i)\log(1 - p_i)\Big]}\]
This is the negative log-likelihood, also called cross-entropy or log loss. Each term is \(-\log(\text{probability given to the truth})\): near 0 when confident and right, huge when confident and wrong (\(-\log 0.01 = 4.6\)).
Least squares is itself maximum likelihood when the errors are Gaussian, so maximum likelihood is the general principle and week 2 was a special case.
A point has \(y = 1\) but the model gives \(p = 0.01\). How much does it add to the NLL?
\(-\log 0.01 = 4.605\). With \(p = 0.99\) it would add only \(0.01\). Confident mistakes are punished hard.
3. The gradient and gradient descent
Setting \(\nabla L = 0\) has no algebraic solution (\(\beta\) is trapped inside the sigmoid), so we use gradient descent. Chain rule on one point, \(\beta \to t_i \to p_i \to L_i\):
\[\frac{\partial L_i}{\partial p_i} = -\Big[\frac{y_i}{p_i} - \frac{1 - y_i}{1 - p_i}\Big],\quad \frac{\partial p_i}{\partial t_i} = p_i(1 - p_i),\quad \frac{\partial t_i}{\partial\beta} = \tilde x_i.\]
Multiply the first two and everything cancels to \(-(y_i - p_i)\). So
\[\boxed{\nabla L(\beta) = -\sum_{i=1}^{n}(y_i - p_i)\,\tilde x_i = -\tilde{\mathbf X}^T(\mathbf y - \mathbf p)}\]
Same shape as the RSS gradient: each point pushes along its own \(\tilde x_i\) in proportion to how wrong it was. The NLL is convex, so gradient descent with a small step finds the minimum.
Worked example: one step by hand. \(x = (-1, 0, 1, 2)\), \(y = (0, 0, 1, 1)\), start \(\beta = (0, 0)\), \(\eta = 0.1\).
All \(p_i = 0.5\), \(L = 4\log 2 = 2.773\). \(y - p = (-0.5, -0.5, 0.5, 0.5)\).
\(\partial L/\partial\beta_0 = -\sum(y_i - p_i) = 0\). \(\partial L/\partial\beta_1 = -\sum(y_i - p_i)x_i = -(0.5 + 0 + 0.5 + 1) = -2\).
\(\beta \leftarrow (0, 0) - 0.1(0, -2) = (0, 0.2)\). New loss \(2.402\): lower. (These classes are perfectly separable at \(x = 0.5\), so \(\beta_1\) would grow forever; an \(L_2\) penalty, lab Q2, stops that.)
One point, \(x = 2\), \(y = 1\). At \(\beta = (0, 0)\) its gradient is \(-(y - p)\tilde x\) with \(p = 0.5\). After one step with \(\eta = 0.1\), what is \(\beta_1\)?
Gradient \(-(0.5)(1, 2) = (-0.5, -1)\). \(\beta = (0, 0) - 0.1(-0.5, -1) = (0.05, 0.1)\), so \(\beta_1 = 0.1\).
4. More than two classes: softmax
Give each class \(k\) its own linear score \(z_k = \beta_{k0} + \sum_j\beta_{kj}x_j\) (so \((p + 1) \times m\) coefficients), and turn the scores into probabilities:
\[\hat p_k(x) = \frac{e^{z_k}}{\sum_{k'}e^{z_{k'}}}.\]
Each \(e^{z_k}\) is positive and dividing by the total makes them sum to 1; a bigger score always gets a bigger probability, so the predicted class is the argmax of the scores. Adding the same constant to every score changes nothing (that’s why code subtracts \(\max z\) first). With two classes, softmax is the sigmoid of the difference of the scores.
Worked example. Scores \((2, 1, 0)\): \(e^2 = 7.389\), \(e^1 = 2.718\), \(e^0 = 1\), total 11.107, probabilities \((0.665, 0.245, 0.090)\). Predict class 1 (the first).
Stage test
(4 marks, exam format 2; 2023 Q3.) Write down the objective function minimised to train a logistic regression model on \(D = \{(X_n, Y_n)\}_{n=1}^N\) with \(Y_n \in \{0, 1\}\), and explain where it comes from.
With \(p_n = P(Y = 1 \mid X_n) = \sigma(\lambda_0 + \lambda_1X_{n1} + \dots)\), the likelihood of the observed labels is \(\ell(\lambda) = \prod_{n=1}^N p_n^{Y_n}(1 - p_n)^{1 - Y_n}\) (each factor is the probability the model gives to what actually happened, assuming independent points). Maximising it is the same as minimising the negative log-likelihood \(L(\lambda) = -\sum_{n=1}^N\big[Y_n\log p_n + (1 - Y_n)\log(1 - p_n)\big]\), because log is increasing; the log turns the product into a sum, which is numerically stable and easy to differentiate. There is no closed-form minimiser, so \(\lambda\) is found by gradient descent, using \(\nabla L = -\sum_n(Y_n - p_n)\tilde X_n\).
Why can’t logistic regression be fitted with a formula like \((\mathbf X^T\mathbf X)^{-1}\mathbf X^T\mathbf y\)?
The NLL is convex (one minimum), but it has no algebraic minimiser. Gradient descent finds it.
How many coefficients does multinomial (softmax) logistic regression have with \(m = 3\) classes and \(p = 4\) inputs?
\((p + 1)\times m = 5 \times 3 = 15\).
In the lab, grad_log_likelihood returns np.sum(A) where A has shape \(n \times 3\). What goes wrong?
A quick numerical-gradient check (nudge each \(\beta_j\) by \(\pm h\)) catches this kind of bug.