Neural network classifier
Give the network one output per class, turn the outputs into probabilities with softmax, and train it to give the true class a high probability (cross-entropy). It's multinomial logistic regression with learned features.
Question type: classification.
Week: 4.
Book: ISLP 10.2 (p405), 10.1.
Techniques on this page: sigmoid output for two classes, one output per class plus softmax, one-hot labels, cross-entropy, the \(p - y\) error signal, the lab’s softmax bug.
The problem it solves
Logistic regression draws straight-line boundaries between classes. Handwritten digits aren’t separated by straight lines in pixel space. A neural network bends: its hidden layer invents features, and a logistic-regression-style output on top of those features can separate classes with curved boundaries.
Everything from week 3 carries over (sigmoid, softmax, negative log-likelihood, argmax). The only new part is the hidden layer underneath.
Words you’ll hear
| Word | Plain meaning |
|---|---|
| logits \(f_m(X)\) | the raw output scores, one per class, before softmax |
| softmax | turns scores into probabilities that add to 1: \(e^{f_m}/\sum e^{f_{m'}}\) |
| one-hot | the label as a 0/1 vector with a single 1: class 2 of 4 is \((0, 0, 1, 0)\) |
| cross-entropy | \(-\log(\text{probability given to the true class})\), added over points |
nn.CrossEntropyLoss |
PyTorch’s softmax + cross-entropy in one: takes logits |
Technique 1: two classes, a sigmoid on top
Problem it solves: a yes/no answer as a probability.
\(P(Y = 1 \mid X) \approx \sigma(f(X))\) and train with week 3’s negative log-likelihood. With no hidden layer this is logistic regression.
Technique 2: \(M\) classes, \(M\) outputs and softmax
Problem it solves: probabilities for every class from one network.
The hidden layer is shared; each class gets its own output \(f_m = \beta_{m0} + \sum_k\beta_{mk}A_k\). Then
\[p_m(X) = \frac{e^{f_m(X)}}{\sum_{m'}e^{f_{m'}(X)}},\qquad \hat Y = \arg\max_m p_m(X).\]
Softmax keeps the order, so the argmax of the logits gives the same prediction. Parameters: \(K(p + 1) + M(K + 1)\).
Technique 3: cross-entropy and its gradient
Problem it solves: training with probabilities, not squared errors.
\[\ell = -\sum_i\sum_m y_{im}\log p_m(x_i) = -\sum_i\log p_{y_i}(x_i)\]
The one-hot \(y_{im}\) picks out the true class. Its gradient at the outputs is simply
\[\frac{\partial\ell_i}{\partial f_m} = p_m(x_i) - y_{im}:\]
predicted minus truth, exactly like logistic regression. Backprop then carries this into the hidden layer.
Worked example. Logits \((2, 1, 0)\), true class 0.
- \(e^2 = 7.389\), \(e^1 = 2.718\), \(e^0 = 1\); total 11.107. \(p = (0.665, 0.245, 0.090)\).
- Loss \(-\log 0.665 = 0.408\). (If the truth were class 2: \(-\log 0.090 = 2.41\).)
- Gradient \(p - y = (-0.335, 0.245, 0.090)\): push class 0’s score up, the others down.
Technique 4: logits in, softmax inside the loss
Problem it solves: the lab’s network can’t learn properly.
Lab bug: ShallowNet.forward ends with nn.Softmax, and then nn.CrossEntropyLoss applies softmax again. The outputs feeding the loss are squashed into \([0, 1]\), so the loss can never drop below \(\log(1 + 9/e) \approx 1.461\) for 10 classes. That’s why the lab’s training loss stalls at about 1.465. Predictions still look fine (softmax keeps the argmax), but the loss values are meaningless and the gradients are weakened. Fix: return self.fc2(self.act(self.fc1(x))) (logits) and use torch.softmax(logits, dim=1) only when you want probabilities. In the practice notebook the fixed version reaches a training loss below 0.01.
What to read in the book
| Section | Read? | Why |
|---|---|---|
| 10.2, p405 (from “We now get to the output layer”) | Read | Softmax (10.13), one-hot, cross-entropy (10.14) |
| 10.2, Table 10.1 | Skim | Networks vs multinomial logistic regression and LDA on MNIST |
| 4.3.5 Multinomial logistic regression (week 3) | Revise | The same softmax |
Practice prompt
I'm studying CS5100 Data Analysis (textbook ISLP, sections 10.1-10.2). I want practice questions on NEURAL NETWORKS FOR CLASSIFICATION, done by hand:
- two classes: a sigmoid on the network output and the negative log-likelihood
- softmax of a vector of logits (3-4 classes, friendly numbers)
- one-hot encoding and the cross-entropy loss for one point and for a few points
- the gradient p - y at the outputs, and which way gradient descent pushes each logit
- counting parameters with M outputs
- why applying softmax before a loss that already applies softmax is a bug, and the minimum loss it causes
- how a neural network classifier relates to multinomial logistic regression
Please:
1. Use plain, everyday language and explain any technical term in one short sentence.
2. Give me ONE question at a time, easy first, then harder, with friendly numbers.
3. Wait for my answer, then mark it and show the full working step by step.
4. If I get it wrong, explain the mistake and give me a similar question.
5. Every 5 questions, summarise my strengths and what to practise.