Logistic regression: the model

2023 Q3 was all logistic regression: the sigmoid’s domain and range, a “show that \(\sigma' = \sigma(1 - \sigma)\)” proof, the likelihood used for training, and a prediction with \(\lambda = [1, 1, -1]\). This stage covers everything except training (that’s stage 10).

1. Why not just fit a line?

Classes are coded \(Y \in \{0, \dots, m - 1\}\), but the codes are labels, not numbers: brown = 0, green = 1, blue = 2 is a free choice. Two problems with linear regression on codes:

1. More than two classes: a line assumes green is “halfway” between brown and blue. Renumber the classes and you get a different fit and different predictions from the same data.
2. Two classes: a line fitted to 0/1 data can be read as an estimate of \(P(Y = 1 \mid X)\), but nothing stops it going below 0 or above 1.

Every classifier this week follows the same recipe: estimate \(P(Y = k \mid X = x)\) for each class, then apply the Bayes classifier rule from stage 4: pick the biggest.

2. The sigmoid

Keep the linear part \(t = \beta_0 + \beta_1X\) (any real number) and squash it into \((0, 1)\) with the sigmoid (logistic) function:

\[\sigma(t) = \frac{1}{1 + e^{-t}} = \frac{e^t}{1 + e^t}, \qquad p(X) = P(Y = 1 \mid X) = \sigma(\beta_0 + \beta_1X).\]

Properties to know for the exam:

• Domain: all real numbers. Range: strictly between 0 and 1. It never reaches 0 or 1, because \(e^{-t} > 0\) makes the denominator bigger than 1 and finite.
• \(\sigma(0) = 0.5\); large positive \(t\) gives nearly 1; large negative \(t\) nearly 0.
• Symmetry: \(\sigma(-t) = 1 - \sigma(t)\).
• It’s increasing, so \(\sigma(t) > 0.5\) exactly when \(t > 0\).

\(t\)	\(-3\)	\(-2\)	\(-1\)	\(0\)	\(1\)	\(2\)	\(3\)
\(\sigma(t)\)	0.047	0.119	0.269	0.5	0.731	0.881	0.953

3. Odds, log-odds, and the derivative

The odds are \(\frac{p}{1 - p}\). For the logistic model, \(1 - p = \frac{1}{1 + e^t}\), so

\[\frac{p}{1 - p} = e^{t}, \qquad \log\frac{p}{1 - p} = \beta_0 + \beta_1X.\]

The log-odds (logit) are linear in \(X\): that’s the sense in which logistic regression is “linear”. Increasing \(X\) by 1 adds \(\beta_1\) to the log-odds, so it multiplies the odds by \(e^{\beta_1}\). The change in \(p\) itself is not fixed: it depends on where you are on the S-curve.

The derivative (2023 Q3c, a “show that”).

\[\sigma'(t) = \frac{d}{dt}(1 + e^{-t})^{-1} = -(1 + e^{-t})^{-2}\cdot(-e^{-t}) = \frac{e^{-t}}{(1 + e^{-t})^2} = \underbrace{\frac{1}{1 + e^{-t}}}_{\sigma(t)}\cdot\underbrace{\frac{e^{-t}}{1 + e^{-t}}}_{1 - \sigma(t)}.\]

The last factor is \(1 - \sigma(t)\) because \(1 - \frac{1}{1 + e^{-t}} = \frac{e^{-t}}{1 + e^{-t}}\). So \(\boxed{\sigma'(t) = \sigma(t)\big(1 - \sigma(t)\big)}\). Check at \(t = 0\): \(0.5 \times 0.5 = 0.25\), the steepest point.

By the chain rule, \(\frac{dp}{dX} = p(1 - p)\beta_1\): biggest (\(0.25\beta_1\)) where \(p = 0.5\), tiny near 0 or 1.

4. Predicting, and several inputs

With \(p\) inputs, put a 1 in front of \(x\) and write \(t = \tilde x^T\beta = \beta_0 + \sum_j\beta_jx_j\); then \(p(x) = \sigma(t)\). Predict class 1 when \(\sigma(t) \ge 0.5\), i.e. when \(t \ge 0\). So the decision boundary \(\beta_0 + \beta_1x_1 + \beta_2x_2 = 0\) is a straight line in the input plane.

Everything from week 2 carries over: dummy variables, polynomial features, penalties. \(\beta_j\) is the change in log-odds per unit of \(x_j\) holding the others fixed. The student paradox (ISLP 4.3.4): alone, student has a positive coefficient (students default more); with balance in the model it turns negative (at the same balance, students default less). Students carry bigger balances. Same story as stage 6’s newspaper example.

Stage test