Maths toolkit
Fitting a model means finding the bottom of an error valley. For that you need vectors (to hold the data and parameters) and derivatives (to find the bottom). This stage is all the maths weeks 1 to 4 lean on.
You’ll practise the operations that appear inside exam calculations: dot products (the 2023 orthogonality question), Euclidean distance written for any \(n\) (the 2023 KNN question), derivatives with the chain rule (the 2023 sigmoid-derivative proof), and setting a derivative to zero to minimise a sum of squared errors (the whole of week 2).
Book: what to read for this stage
ISLP chapter 1, Notation and Simple Matrix Algebra (two pages). The calculus on the week 1 slides (“Linear algebra and calculus basics”) goes beyond the book, so this stage is the reference. Nothing else in the book is needed here.
1. Vectors, the data matrix and the dot product
A data point is a vector \(x_i = (x_{i1}, \dots, x_{ip})\): one number per input. Stack the \(n\) data points as rows and you get the data matrix \(\mathbf X\), with \(n\) rows and \(p\) columns. Row \(i\) is one observation; column \(j\), written \(\mathbf x_j\), is every observation of input \(j\). In pandas, a DataFrame is \(\mathbf X\): df.shape == (n, p).
The transpose \(^T\) turns a row into a column (and a matrix’s rows into its columns).
For two vectors of the same length, the dot (inner) product is a single number:
\[a \cdot b = a^Tb = \sum_{j=1}^{p} a_jb_j.\]
Two vectors are orthogonal (at right angles) exactly when \(a \cdot b = 0\).
The outer product \(ab^T\) is a \(p \times p\) matrix with entries \(a_jb_k\). In NumPy, a @ b is the dot product, np.outer(a, b) the outer product, and a * b is elementwise, which is neither.
Worked example (2023 Q1c). Are \((1, 2, -3)\) and \((5, -4, -1)\) orthogonal?
\(1 \times 5 + 2 \times (-4) + (-3) \times (-1) = 5 - 8 + 3 = 0\). The dot product is 0, so yes.
And for \(a = (1, 2, 3)\), \(b = (4, 5, 6)\): \(a \cdot b = 4 + 10 + 18 = 32\), while a * b \(= (4, 10, 18)\).
A data set has 392 cars and 7 numeric variables. What shape is \(\mathbf X\), and how long is the column \(\mathbf x_3\)?
Rows are observations (\(n = 392\)), columns are variables (\(p = 7\)). A column holds one variable for every observation, so it has \(n = 392\) entries. Mixing up rows and columns is the most common notation slip.
2. Lengths and distances
The Euclidean length of a vector is \(\|v\| = \sqrt{v \cdot v} = \sqrt{\sum_j v_j^2}\), and the distance between two points in \(\mathbb R^n\) is the length of their difference:
\[d(x, x') = \|x - x'\| = \sqrt{\sum_{i=1}^{n}(x_i - x'_i)^2}.\]
This is the distance KNN uses. The 2023 paper asked you to “write a distance measure in \(\mathbb R^n\) showing the dependence on \(n\)”: write exactly this, with the sum running to \(n\).
More generally the \(p\)-norm is \(\|v\|_p = \big(\sum_j |v_j|^p\big)^{1/p}\) (week 2 lab). \(p = 2\) is Euclidean; \(p = 1\) is the sum of absolute values.
Worked example. \(x = (1, 2, 2)\), \(x' = (4, -2, 2)\). Differences \((-3, 4, 0)\), squares \(9, 16, 0\), so \(d = \sqrt{25} = 5\). The 1-norm of the difference is \(3 + 4 + 0 = 7\).
Compute the 1-norm \(\|(3, -4)\|_1\). (The 2-norm is 5.)
\(|3| + |-4| = 7\).
3. Derivatives, and the rules you need
The derivative \(f'(z)\) is the slope of \(f\) at \(z\): how much \(f\) rises per unit step, for a tiny step.
\[f'(z) = \lim_{\delta \to 0}\frac{f(z + \delta) - f(z)}{\delta}.\]
Slide example, \(f(z) = z^2\): \(\frac{(z + \delta)^2 - z^2}{\delta} = \frac{2z\delta + \delta^2}{\delta} = 2z + \delta \to 2z\).
| Rule | Formula |
|---|---|
| power | \((z^r)' = rz^{r-1}\) |
| exp and log | \((e^z)' = e^z\), \(\quad(\log z)' = 1/z\) (log means natural log) |
| sum, constant | \((f + g)' = f' + g'\), \(\quad(cf)' = cf'\) |
| product | \((fg)' = f'g + fg'\) |
| chain | \(\big(f(g(z))\big)' = f'(g(z))\,g'(z)\): multiply the slopes along the chain |
Worked example: the chain rule four ways.
- \((e^{-z^2})' = e^{-z^2} \cdot (-2z)\).
- \((\log(1 + z^2))' = \frac{1}{1 + z^2} \cdot 2z\).
- \(((5z - 2)^3)' = 3(5z - 2)^2 \cdot 5 = 15(5z - 2)^2\).
- \(\big(\frac{1}{1 + e^{-z}}\big)'\): the outer \(u^{-1}\) has slope \(-u^{-2}\), the inner \(u = 1 + e^{-z}\) has slope \(-e^{-z}\). Product: \(\frac{e^{-z}}{(1 + e^{-z})^2}\). That’s the sigmoid’s derivative, which you’ll rewrite as \(\sigma(1 - \sigma)\) in stage 9.
What is the derivative of \(f(z) = \log(1 + e^{z})\)?
Chain rule: the outer \(\log u\) has slope \(1/u\), the inner \(u = 1 + e^z\) has slope \(e^z\). Multiply: \(\frac{e^z}{1 + e^z}\), which is the sigmoid \(\sigma(z)\).
For \(f(z) = z^4 - 2z + 7\), give \(f'(1)\).
\(f'(z) = 4z^3 - 2\) (power rule; the constant 7 has slope 0), so \(f'(1) = 4 - 2 = 2\).
4. Partial derivatives, gradients, and finding a minimum
If \(f\) depends on several numbers \(z = (z_1, \dots, z_p)\), the partial derivative \(\partial_{z_j}f\) is the ordinary derivative in \(z_j\) with every other \(z\) held constant. The gradient stacks them: \(\nabla f = (\partial_{z_1}f, \dots, \partial_{z_p}f)^T\). It points uphill; minus the gradient points downhill.
At the bottom of a smooth valley every slope is zero. So to minimise an error, set its derivative (or gradient) to zero and solve.
Worked example (the slides’ key example). Fit a line through the origin, \(\hat y = zx\), by minimising \(g(z) = \sum_{i=1}^n (zx_i - y_i)^2\).
\[g'(z) = \sum_i 2(zx_i - y_i)\,x_i \quad\text{(sum rule, then chain rule)}.\]
Set it to zero: \(z\sum_i x_i^2 = \sum_i x_iy_i\), so \(z^* = \dfrac{\sum_i x_iy_i}{\sum_i x_i^2}\).
Data \(x = (1, 2, 3)\), \(y = (2, 3, 7)\): \(\sum x_iy_i = 2 + 6 + 21 = 29\), \(\sum x_i^2 = 14\), so \(z^* = 29/14 \approx 2.071\). It’s a minimum because \(g''(z) = 2\sum x_i^2 = 28 > 0\).
Partial derivatives: \(f(z_1, z_2) = z_1^2z_2 + 3z_2\) has \(\nabla f = (2z_1z_2,\ z_1^2 + 3)\); at \((1, 2)\) that’s \((4, 4)\).
Using the formula above, find the best slope \(z^*\) for a line through the origin fitted to \(x = (1, 2)\), \(y = (3, 4)\).
\(\sum x_iy_i = 3 + 8 = 11\) and \(\sum x_i^2 = 1 + 4 = 5\), so \(z^* = 11/5 = 2.2\).
For \(f(z_1, z_2) = z_1^2z_2 + 3z_2\), what is \(\partial f/\partial z_2\) at \((z_1, z_2) = (1, 2)\)?
Treat \(z_1\) as a constant: \(\partial f/\partial z_2 = z_1^2 + 3 = 1 + 3 = 4\).
Stage test
(3 marks, exam format 2.) Write a formula for the Euclidean distance between two points \(x, x' \in \mathbb R^n\), showing explicitly how it depends on \(n\), and say what each symbol means.
\(d(x, x') = \|x - x'\|_2 = \sqrt{\sum_{i=1}^{n}(x_i - x'_i)^2}\), where \(x = (x_1, \dots, x_n)\) and \(x' = (x'_1, \dots, x'_n)\) are the two points and \(x_i, x'_i\) their \(i\)-th coordinates. Each of the \(n\) dimensions contributes one squared difference, so the distance depends on \(n\) through the number of terms in the sum.
What is \(\frac{d}{dz}\, z e^{2z}\)?
Product rule: \((z)'e^{2z} + z(e^{2z})' = e^{2z} + z \cdot 2e^{2z} = (1 + 2z)e^{2z}\).
For \(f(z_1, z_2, z_3) = z_1z_2 + z_3^2 - 4z_1\), give \(\partial f/\partial z_1\) at the point \((1, 0, 2)\).
\(\partial f/\partial z_1 = z_2 - 4\) (treat \(z_2, z_3\) as constants). At \(z_2 = 0\) that’s \(-4\). The full gradient there is \((-4, 1, 4)\).
For \(L(\beta_0, \beta_1) = \sum_{i=1}^n (y_i - \beta_0 - \beta_1x_i)^2\), which is \(\partial L/\partial\beta_1\)?
Chain rule on each term: the outer square gives \(2(y_i - \beta_0 - \beta_1x_i)\), and the inner bracket has slope \(-x_i\) in \(\beta_1\). This is the simple linear regression loss you’ll minimise in stage 5. The third option is \(\partial L/\partial\beta_0\).
Minimise \(g(z) = \sum_i (zx_i - y_i)^2\) for \(x = (1, 2, 3)\), \(y = (2, 3, 7)\). Give \(z^*\).
\(z^* = \sum x_iy_i / \sum x_i^2 = 29/14 \approx 2.071\).