Training neural networks
There's no formula for the best weights, so start from small random numbers and walk downhill. Backpropagation is the chain rule that tells every weight which way is downhill; PyTorch does it for you with one line.
Question type: not a model: how neural networks are fitted (regression or classification).
Week: 4.
Book: ISLP 10.7, 10.7.1, 10.7.2.
Techniques on this page: non-convex losses, gradient descent from random starts, ridge penalty and early stopping, minibatches and epochs, backpropagation, PyTorch’s training loop.
The problem it solves
For linear regression the error is one smooth bowl and you can solve for the bottom. A network’s weights sit inside the bends, so there’s no formula, and the error surface has many valleys. And with thousands of weights, the network can memorise the training set.
So training is two jobs: get downhill (gradient descent, with backprop computing the gradient), and don’t overfit (penalties, early stopping, validation data).
Words you’ll hear
| Word | Plain meaning |
|---|---|
| non-convex | an error surface with several valleys |
| local minimum | the bottom of one valley, maybe not the lowest |
| learning rate \(\eta\) | step size |
| backpropagation | chain rule, run from the output back to the inputs |
| minibatch, SGD | use a random handful of points for each step |
| epoch | one pass through all the training data |
| early stopping | stop when the validation error starts rising |
| Adam | gradient descent with automatic per-weight step sizes |
| autograd | PyTorch’s automatic chain rule: loss.backward() |
Technique 1: gradient descent from small random starts
Problem it solves: finding good weights without a formula.
Put every weight in one vector \(\theta\) and repeat \(\theta \leftarrow \theta - \eta\nabla R(\theta)\). Start from small random values: if every weight starts at 0, all hidden units are identical and stay identical (and with ReLU nothing moves at all). Different starts can end in different valleys; for big networks, most valleys are good enough.
Technique 2: don’t overfit
Problem it solves: a network with more weights than data points will memorise.
- Ridge penalty: minimise \(\text{RSS} + \lambda\|\theta\|^2\), not penalising the intercepts (week 2’s ridge). In PyTorch,
weight_decay=λ(which does include the biases). - Early stopping (slow learning): watch the validation error each epoch; keep the weights from the best epoch.
- Fewer hidden units.
Technique 3: backpropagation
Problem it solves: computing the gradient for every weight cheaply.
For one point with residual \(e = y - f(x)\) and \(R = e^2\):
\[\frac{\partial R}{\partial\beta_k} = -2e\,A_k,\qquad \frac{\partial R}{\partial w_{kj}} = -2e\cdot\beta_k\cdot g'(z_k)\cdot x_j\]
The error starts at the output and is passed back, shared out by how much each weight contributed. That’s the chain rule; “back” because it runs output → inputs.
Worked example. One input \(x = 2\), target \(y = 3\). Unit 1: \(z_1 = 0 + 1(2) = 2\), \(A_1 = 2\). Unit 2: \(z_2 = 1 - 2 = -1\), \(A_2 = 0\). \(f = 0.5 + 1(2) + 2(0) = 2.5\), \(e = 0.5\), so \(-2e = -1\).
- \(\partial R/\partial\beta_1 = -1 \times 2 = -2\); \(\partial R/\partial\beta_2 = -1 \times 0 = 0\).
- \(\partial R/\partial w_{11} = -1 \times \beta_1 \times 1 \times 2 = -2\).
- Unit 2 gets zero gradient: its ReLU is off (\(z_2 < 0\)). A unit that’s off for every point never learns (“dead”).
- One step with \(\eta = 0.05\) moves \(f\) from 2.5 to 3.025: much closer to 3.
Book vs slides: ISLP puts \(\tfrac12\) in front of the RSS, so its gradients say \(-(y - f)\) where the slides say \(-2(y - f)\). Same direction; only the step size differs. Also, slide eq. (28) sets \(z_{i0} = 1\) to cover the intercept, but the factor for \(\beta_0\) should be 1, not \(g(1)\).
Technique 4: minibatches and epochs
Problem it solves: a full gradient over all the data is slow.
Each step uses a random minibatch (the lab uses 32). One epoch = one pass through the data: 1,257 images in batches of 32 is 40 steps. Noisier steps, much faster progress.
Technique 5: the PyTorch loop
Problem it solves: never writing a gradient by hand again.
for X_b, y_b in train_loader: # one epoch
out = model(X_b) # forward pass
loss = criterion(out, y_b) # nn.MSELoss() or nn.CrossEntropyLoss()
optimizer.zero_grad() # clear old gradients (they add up otherwise)
loss.backward() # backprop: every parameter's .grad is filled in
optimizer.step() # theta <- theta - eta * grad (Adam: smarter steps)Regression shape trap (lab Q2): with nn.MSELoss, the target must be shape \((n, 1)\) like the output. Passing a shape-\((n,)\) target makes PyTorch compare every prediction with every target and silently report the wrong loss. Use y.reshape(-1, 1). Standardising the salary target also makes training far easier.
What to read in the book
| Section | Read? | Why |
|---|---|---|
| 10.7 Fitting a neural network (p427–428) | Read | Non-convexity, slow learning, regularisation |
| 10.7.1 Backpropagation | Read | Mind the \(\tfrac12\) |
| 10.7.2 Regularization and SGD | Read | Minibatches, epochs, early stopping: all in the lab |
| 10.7.3 Dropout, 10.7.4 Network tuning | Skim | Tuning helps with lab Q2 |
| 10.8 Double descent | Skip |
Practice prompt
I'm studying CS5100 Data Analysis (textbook ISLP, section 10.7). I want practice questions on TRAINING NEURAL NETWORKS, done by hand:
- backpropagation for a network with one input, two ReLU or sigmoid hidden units and one output: forward pass, residual, gradients for every weight
- one gradient-descent step and the new prediction
- dead ReLU units and tiny sigmoid slopes
- why we start from small random weights, not zeros
- the ridge penalty's extra gradient term, and early stopping from training/validation curves
- minibatches, epochs and number of steps
- what each line of a PyTorch training loop does, and common bugs (no zero_grad, wrong target shape)
Please:
1. Use plain, everyday language and explain any technical term in one short sentence.
2. Give me ONE question at a time, easy first, then harder, with friendly numbers.
3. Wait for my answer, then mark it and show the full working step by step.
4. If I get it wrong, explain the mistake and give me a similar question.
5. Every 5 questions, summarise my strengths and what to practise.