Neural networks: the model
A neural network invents its own features: each hidden neuron takes a weighted sum of the inputs and bends it with an activation function, then the output is a linear model on those features. You'll count parameters and run forward passes by hand, as 2025 Q2 did.
2025 Q2 (25 marks): “What is a neural network?”, count the parameters of a network with 2 inputs and 5 hidden neurons (and say whether 3 inputs means more), explain why a network is more flexible than a linear model, write an objective function, and do a ReLU forward pass. This stage covers all but the objective (stage 17).
Book: what to read for this stage
ISLP 10.1 Single layer neural networks (eqs. 10.1 to 10.5, sigmoid vs ReLU): read; the \(X_1X_2\) example (eqs. 10.6 to 10.8) is short and worth doing by hand. 10.2 multilayer networks: skim; the slides stay with one hidden layer. 10.3 to 10.6 (CNNs, RNNs, document classification): skip.
1. Hidden neurons and activations
Inputs \(X = (X_1, \dots, X_p)\). A single-hidden-layer network computes:
- Hidden layer: \(K\) new features, each a weighted sum of the inputs plus a bias, pushed through an activation function \(g\): \[A_k = g\Big(w_{k0} + \sum_{j=1}^{p}w_{kj}X_j\Big),\qquad k = 1, \dots, K.\]
- Output: a linear model on the features: \(f(X) = \beta_0 + \sum_{k=1}^{K}\beta_kA_k\).
The \(A_k\) are hidden units (never observed; the network invents them). Information flows one way: feed-forward. In deep-learning language the constants \(w_{k0}\), \(\beta_0\) are called biases (nothing to do with bias-variance).
The two activations on the slides:
- Sigmoid \(\sigma(z) = 1/(1 + e^{-z})\): squashes into \((0, 1)\) (“firing” near 1, “silent” near 0).
- ReLU \(\max(0, z)\): negatives become 0, positives pass through. Cheap, and its slope is 1 for every positive \(z\), so it doesn’t stall training the way the sigmoid’s flat tails do. The modern default.
Compare week 2’s basis functions: there you chose the new columns (\(X^2\), \(X^3\)); here the network learns them, because the \(w\)’s are fitted too.
Worked example. Two inputs, two ReLU units: unit 1 has \((w_{10}, w_{11}, w_{12}) = (0, 1, 1)\), unit 2 has \((-1, 1, -1)\); output \(\beta = (1, 2, -1)\); input \(x = (1, 2)\).
- \(z_1 = 0 + 1 + 2 = 3\), \(A_1 = 3\).
- \(z_2 = -1 + 1 - 2 = -2\), \(A_2 = \mathrm{ReLU}(-2) = 0\) (silent).
- \(f(x) = 1 + 2 \times 3 - 1 \times 0 = 7\).
With sigmoid instead: \(A = (0.953, 0.119)\), \(f = 1 + 1.905 - 0.119 = 2.786\).
2. Counting parameters
Each hidden neuron has one weight per input plus a bias: \(K(p + 1)\). The output has one weight per hidden neuron plus a bias: \(K + 1\) (or \(M(K + 1)\) with \(M\) outputs). Total:
\[K(p + 1) + (K + 1).\]
2025 Q2b: 2 inputs, 5 hidden neurons, 1 output: \(5 \times 3 + 6 = 21\). With 3 inputs: \(5 \times 4 + 6 = 26\), so yes, more inputs means more parameters (5 more: one per hidden neuron). ISLP Fig. 10.1 (\(p = 4\), \(K = 5\)): \(25 + 6 = 31\). PyTorch’s nn.Linear(p, K) stores a \(K \times p\) weight matrix and \(K\) biases.
3. Why the activation must be non-linear, and why networks are flexible
If \(g(z) = z\), substitute and collect terms:
\[f(X) = \Big(\beta_0 + \sum_k\beta_kw_{k0}\Big) + \sum_j\Big(\sum_k\beta_kw_{kj}\Big)X_j,\]
which is just a linear model, however many hidden units you add. The non-linear \(g\) is what lets the network bend. ISLP’s example: with \(g(z) = z^2\), units \((1, 1)\) and \((1, -1)\) and \(\beta = (0, \frac14, -\frac14)\), \(f = \frac14(X_1 + X_2)^2 - \frac14(X_1 - X_2)^2 = X_1X_2\), an interaction no linear model can produce. With enough hidden units, sigmoid or ReLU networks can approximate essentially any smooth function.
With one input, each ReLU unit is a hinge; \(f\) adds up \(K\) hinges, a piecewise-linear curve with up to \(K\) bends. More hidden units, more bends: \(K\) is the flexibility knob (lower bias, higher variance, more risk of overfitting). Compare a tree, which is piecewise constant.
A network with 1,000 hidden units uses the identity activation \(g(z) = z\). How flexible is it?
A composition of linear maps is linear. The non-linearity is what makes hidden layers worth having.
4. The 2025 forward pass
Worked example (2025 Q2e). Five hidden ReLU neurons with \(\lambda_i = [i, 1, \tfrac12]\) for \(i = 1, \dots, 5\) (bias \(i\), weights \(1\) and \(\frac12\)), output \(\lambda_o = [-5, 4, 3, 2, 1, 0]\) with a ReLU on the output too, input \(X = [-1, -2]\).
- \(Z_i = i + 1 \cdot(-1) + \tfrac12 \cdot(-2) = i - 2\), so \(Z = (-1, 0, 1, 2, 3)\).
- After ReLU: \((0, 0, 1, 2, 3)\).
- Output: \(\mathrm{ReLU}(-5 + 4 \cdot 0 + 3 \cdot 0 + 2 \cdot 1 + 1 \cdot 2 + 0 \cdot 3) = \mathrm{ReLU}(-1) = \mathbf 0\).
Read each question’s definition: here the output neuron also had a ReLU; in the slides’ regression network the output is linear.
Same network as 2025, but with input \(X = [1, 0]\). Compute the output.
\(Z_i = i + 1 + 0 = i + 1\), so \(Z = (2, 3, 4, 5, 6)\), all positive, ReLU leaves them. Output \(\mathrm{ReLU}(-5 + 4 \cdot 2 + 3 \cdot 3 + 2 \cdot 4 + 1 \cdot 5 + 0 \cdot 6) = \mathrm{ReLU}(-5 + 8 + 9 + 8 + 5) = 25\).
Stage test
(5 marks, exam format 1; 2025 Q2a.) What is a neural network? Describe the input, hidden and output neurons of a single-hidden-layer network, and give an example of a task it could be used for.
A neural network is a flexible non-linear model that builds its prediction in layers. In a single-hidden-layer (feed-forward) network the input neurons simply hold the \(p\) inputs \(X_1, \dots, X_p\). Each of the \(K\) hidden neurons computes a weighted sum of all the inputs plus a bias and passes it through a non-linear activation function: \(A_k = g(w_{k0} + \sum_jw_{kj}X_j)\), with \(g\) = ReLU \(\max(0, z)\) or the sigmoid. The output neuron combines the hidden activations linearly, \(f(X) = \beta_0 + \sum_k\beta_kA_k\) (for classification, followed by a sigmoid or softmax). All the weights and biases are learned from training data by gradient descent. Example: predicting a house’s price (output) from its floor area, number of rooms and distance to the centre (inputs); or recognising the digit in an \(8 \times 8\) image (64 inputs, 10 outputs).
2025 Q2b: how many parameters does a network with 2 inputs, 5 hidden neurons and 1 output have (every neuron with a bias)?
Hidden: \(5 \times (2 + 1) = 15\). Output: \(5 + 1 = 6\). Total 21.
(3 marks, exam format 3; 2025 Q2.) Why is a neural network more flexible than a linear model?
A linear model can only produce \(\lambda_0 + \sum_j\lambda_jX_j\), a flat plane. A network passes weighted sums of the inputs through non-linear activations (ReLU, sigmoid) in its hidden layer and then combines them, so \(f\) is a non-linear function that can bend, have corners and represent interactions such as \(X_1X_2\); in effect it learns its own basis functions from the data, and with enough hidden neurons it can approximate essentially any smooth function. (If the activation were the identity, the network would collapse back to a linear model.) The price is many more parameters: more hidden neurons give lower bias but higher variance, so networks can overfit and need regularisation or early stopping.
A ReLU hidden unit has \(z = -2.5\) for some input. What is its activation and what does it contribute to the output for that input?
\(\max(0, -2.5) = 0\), so \(\beta_k \cdot 0 = 0\). (0.076 would be the sigmoid.)