Neural network (one hidden layer)
Invent K new features, each a weighted sum of the inputs pushed through a simple bend (ReLU or sigmoid), then fit a straight-line model on those features. The network learns which features to invent.
Question type: regression here; the classifier version has its own page.
Week: 4.
Book: ISLP 10.1.
Techniques on this page: hidden units and activations, sigmoid and ReLU, why the bend is essential, counting parameters, the RSS objective. Fitting it is on Training neural networks.
The problem it solves
In week 2 you bent a straight line by inventing new columns yourself: \(x^2\), \(x^3\), steps (polynomial and step regression). That works if you can guess the right columns. With 64 pixel inputs, who knows what the right columns are?
A neural network learns its own new columns. Each hidden unit is a weighted sum of the inputs with a bend applied; the output is linear regression on those units. With enough units it can approximate almost any smooth relationship, including interactions between inputs, without you designing anything. The cost: many parameters, no formula for fitting them, and a real risk of overfitting.
Words you’ll hear
| Word | Plain meaning |
|---|---|
| hidden unit, neuron, activation \(A_k\) | one invented feature: \(g(\text{weighted sum of inputs})\) |
| hidden layer | all \(K\) hidden units together |
| weights \(w_{kj}\), bias \(w_{k0}\) | the multipliers and the constant inside a hidden unit (“bias” here just means intercept) |
| activation function \(g\) | the fixed bend: sigmoid or ReLU |
| ReLU | \(\max(0, z)\): negatives become 0, positives pass through |
| feed-forward | information flows inputs → hidden → output, never back |
nn.Linear(p, K) |
PyTorch’s “\(K\) weighted sums of \(p\) inputs, plus biases” |
Technique 1: hidden units
Problem it solves: building useful new features automatically.
\[A_k = g\Big(w_{k0} + \sum_{j=1}^{p}w_{kj}X_j\Big),\quad k = 1, \dots, K \qquad\qquad f(X) = \beta_0 + \sum_{k=1}^{K}\beta_kA_k\]
Every input feeds every hidden unit; the output is a linear model in the \(A_k\).
Worked example (forward pass). Two inputs, two ReLU units. Unit 1: bias 0, weights \((1, 1)\). Unit 2: bias \(-1\), weights \((1, -1)\). Output: \(\beta_0 = 1\), \(\beta = (2, -1)\). Input \(x = (1, 2)\).
- \(z_1 = 0 + 1 + 2 = 3\), so \(A_1 = 3\).
- \(z_2 = -1 + 1 - 2 = -2\), so \(A_2 = 0\) (silent).
- \(f = 1 + 2(3) - 1(0) = 7\).
Technique 2: the bend (activation functions)
Problem it solves: without it, the network is just linear regression.
- Sigmoid \(\sigma(z) = 1/(1 + e^{-z})\): squashes into \((0, 1)\), the “neuron firing or silent” picture. Slope at most 0.25, nearly 0 for large \(|z|\).
- ReLU \(\max(0, z)\): the modern default. Cheap, and slope exactly 1 for every positive \(z\).
Why the bend matters. If \(g(z) = z\), substituting gives \(f = \tilde\beta_0 + \sum_j\tilde\beta_jX_j\) with \(\tilde\beta_j = \sum_k\beta_kw_{kj}\): a linear model, however many units. In the example that’s \(2 + x_1 + 3x_2\). With a bend, two units can even make an interaction: with \(g(z) = z^2\), \(\frac14(X_1 + X_2)^2 - \frac14(X_1 - X_2)^2 = X_1X_2\) (ISLP p402).
With one input and ReLU, each unit is a hinge, flat then sloped. \(f\) adds \(K\) hinges: a bendy line with up to \(K\) corners. More units, more corners. (A tree makes steps; a ReLU network makes ramps.)
Technique 3: counting parameters
Problem it solves: knowing how big (and how overfit-prone) a network is.
Hidden layer: \(K(p + 1)\). Output: \(K + 1\). ISLP Fig 10.1 with \(p = 4\), \(K = 5\): \(25 + 6 = 31\). The lab’s digits net (\(p = 64\), \(K = 128\), 10 outputs): \(128 \times 65 + 10 \times 129 = 9{,}610\).
Compare with the number of training points: a network with more parameters than data points can memorise them.
Technique 4: the objective
Problem it solves: saying what “a good fit” is.
For a numeric response, minimise the RSS, \(\sum_i(y_i - f(x_i))^2\), as in linear regression. The weights sit inside \(g\), so there’s no formula: you walk downhill with gradient descent and backpropagation (Training neural networks).
Network vs the other models
| linear regression | polynomial regression | regression tree | neural network | |
|---|---|---|---|---|
| new features | none | you choose (\(x^2\), …) | boxes | learned |
| shape (1 input, ReLU) | line | smooth curve | steps | ramps with corners |
| fitting | formula | formula | greedy splits | gradient descent |
| readable | yes | so-so | yes | no |
What to read in the book
| Section | Read? | Why |
|---|---|---|
| 10.1 Single layer neural networks (p400–402) | Read | Core. Includes the \(X_1X_2\) example |
| 10.2 Multilayer networks (p402–406) | Skim, except p405 | The slides use one hidden layer; p405 is the classifier part |
| 10.3–10.6 CNNs, RNNs, when to use | Skip | Not on the slides |
Practice prompt
I'm studying CS5100 Data Analysis (textbook ISLP, section 10.1). I want practice questions on a SINGLE-HIDDEN-LAYER NEURAL NETWORK for regression, done by hand:
- forward passes with 2 inputs and 2-3 hidden units, using ReLU and using the sigmoid
- what happens when the activation is g(z) = z: collapse to linear regression and find the equivalent coefficients
- how two hidden units with g(z) = z^2 can produce an interaction X1*X2
- what one ReLU unit looks like as a function of one input (a hinge), and what a sum of hinges looks like
- counting parameters for given p, K and number of outputs
- comparing a neural network with polynomial regression and with a regression tree
Please:
1. Use plain, everyday language and explain any technical term in one short sentence.
2. Give me ONE question at a time, easy first, then harder, with friendly numbers.
3. Wait for my answer, then mark it and show the full working step by step.
4. If I get it wrong, explain the mistake and give me a similar question.
5. Every 5 questions, summarise my strengths and what to practise.