Maths toolkit
Not a model. These are the tools the models borrow. Each one says which model needs it, so you know why you're learning it.
Why the week 1 slides suddenly do calculus
Fitting a model means finding the numbers that make its error as small as possible. “Make something as small as possible” is exactly what calculus is for: at the bottom of a valley the ground is flat, and calculus measures how steep the ground is. That’s the only reason it’s here. You’ll use it properly in linear regression (week 2) and logistic regression (week 3).
1. Data as a grid of numbers (vectors and matrices)
Used by: every model.
A spreadsheet of data is called the data matrix \(\mathbf{X}\). It has \(n\) rows (people) and \(p\) columns (pieces of information).
\[\mathbf{X} = \begin{pmatrix} x_{11} & \dots & x_{1p}\\ \vdots & \ddots & \vdots\\ x_{n1} & \dots & x_{np}\end{pmatrix}\]
- A row \(x_i\) is one person: all their information.
- A column \(\mathbf{x}_j\) is one piece of information for everyone.
- A vector is just a list of numbers. A matrix is a grid of numbers.
- The transpose \(^T\) flips a row into a column (or a grid on its side). It’s bookkeeping, nothing deep.
In code, a pandas DataFrame is \(\mathbf{X}\), and df.shape gives \((n, p)\).
The dot product
Used by: linear regression, logistic regression, correlation.
Problem it solves: “multiply each input by its weight and add it all up” happens constantly. The dot product is the shorthand.
\[a \cdot b = a^T b = a_1 b_1 + a_2 b_2 + \dots + a_p b_p\]
\(a = (1, 2, 3)\), \(b = (4, 5, 6)\): \(\ a\cdot b = 4 + 10 + 18 = 32\).
In NumPy that’s a @ b. Careful: a * b gives \((4, 10, 18)\), multiplying piece by piece without adding.
2. The derivative: how steep is it here?
Used by: linear regression (finding the best line), logistic regression, gradient descent.
The derivative \(f'(z)\) is the slope of the graph of \(f\) at the point \(z\): how much \(f\) goes up if you nudge \(z\) up a tiny bit. You can always estimate it with a calculator:
\[f'(z) \approx \frac{f(z + h) - f(z)}{h}\quad\text{for a tiny } h\]
The slides write the exact version with a limit, \(\lim_{\delta\to 0}\), which just means “make the nudge smaller and smaller and see what the answer settles on.”
\(f(z) = z^2\) at \(z = 3\), nudge \(h = 0.0001\): \(\ \dfrac{3.0001^2 - 9}{0.0001} \approx 6.0001\). The exact rule below gives \(2z = 6\).
The slides’ version done exactly: \(\dfrac{(z+\delta)^2 - z^2}{\delta} = \dfrac{2z\delta + \delta^2}{\delta} = 2z + \delta\), and as \(\delta\) shrinks to 0 that’s \(2z\).
Why we care: at the lowest point of a valley the slope is zero. So “find the best numbers” becomes “find where the slope is zero.”
The rules (so you don’t need the limit every time)
| Function | Its slope | Plain words |
|---|---|---|
| a constant \(c\) | \(0\) | flat ground has no slope |
| \(z^r\) | \(r z^{r-1}\) | bring the power down, reduce it by one |
| \(e^z\) | \(e^z\) | the exponential is its own slope |
| \(\log z\) | \(1/z\) | (log means natural log, ln, in this course) |
| \(f + g\) | \(f' + g'\) | slope of a sum = sum of slopes |
| \(f \cdot g\) | \(f'g + fg'\) | product rule |
| \(f(g(z))\) | \(f'(g(z))\cdot g'(z)\) | chain rule, see below |
The chain rule: functions inside functions
Problem it solves: most model errors are “a function of a function” (a square of a difference, a log of a sigmoid). The chain rule says: multiply the slopes along the chain.
\(\dfrac{d}{dz}(5z - 2)^3\). Outside: something cubed, slope \(3(\text{something})^2\). Inside: \(5z - 2\), slope \(5\). Multiply: \(3(5z-2)^2 \cdot 5 = 15(5z-2)^2\).
\(\dfrac{d}{dz}e^{-z^2}\). Outside \(e^{u}\) has slope \(e^u\), inside \(-z^2\) has slope \(-2z\). Answer: \(-2z\,e^{-z^2}\).
3. Several dials at once: partial derivatives and the gradient
Used by: every model with more than one number to fit (which is all of them).
A model usually has several numbers to tune, like \(\beta_0\) and \(\beta_1\). A partial derivative \(\partial f/\partial z_1\) is the slope if you turn only dial \(z_1\) and hold the others still. To compute it, treat the other letters as fixed numbers.
The gradient \(\nabla f\) (“nabla f” or “del f”) is the list of all the partial slopes. It points in the direction that goes uphill fastest. Go the opposite way and you go downhill. That’s the whole idea of gradient descent (week 2).
\(f(z_1, z_2) = z_1^2 z_2 + 3z_2\).
- Turn only \(z_1\) (treat \(z_2\) as a number): \(\partial f/\partial z_1 = 2z_1 z_2\).
- Turn only \(z_2\): \(\partial f/\partial z_2 = z_1^2 + 3\).
- Gradient: \(\nabla f = (2z_1z_2,\ z_1^2 + 3)\). At \((1, 2)\) it’s \((4, 4)\).
4. The slides’ key example: preview of fitting a line
This slide is the bridge to week 2. The model is “guess \(y\) as \(z\) times \(x\)” (a line through the origin with slope \(z\)). Its total squared error is
\[g(z) = \sum_{i=1}^n (z x_i - y_i)^2\]
Slope, by the sum rule and chain rule: \(\ g'(z) = \sum_i 2(z x_i - y_i)\,x_i\).
Set it to zero to find the bottom of the valley:
\[z\sum x_i^2 = \sum x_i y_i \quad\Rightarrow\quad z^* = \frac{\sum x_i y_i}{\sum x_i^2}\]
\(x = (1, 2, 3)\), \(y = (2, 3, 7)\). \(\sum x_iy_i = 2 + 6 + 21 = 29\), \(\sum x_i^2 = 14\), so the best slope is \(29/14 \approx 2.07\).
That is linear regression in miniature. Week 2 does the same with an intercept.
5. Correlation (from the week 1 lab)
Used by: lab Q2, and it reappears inside \(R^2\) in week 2.
Problem it solves: “do these two columns move together, and how strongly?” in a single number from \(-1\) to \(1\).
Centre both lists (subtract their averages) to get \(a\) and \(b\). Then
\[r = \frac{a\cdot b}{\sqrt{a\cdot a}\,\sqrt{b\cdot b}}\]
\(r\) near \(+1\): they rise together in a straight-line way. Near \(-1\): one rises as the other falls. Near 0: no straight-line link (there might still be a curved one).
\(x = (1,2,3,4,5)\), \(y = (2,4,5,4,5)\). Averages 3 and 4. \(a = (-2,-1,0,1,2)\), \(b = (-2,0,1,0,1)\). \(a\cdot b = 6\), \(a\cdot a = 10\), \(b\cdot b = 6\). \(r = 6/\sqrt{60} \approx 0.775\).
6. Practice prompt
I'm studying CS5100 Data Analysis (textbook ISLP). I need practice with the MATHS TOOLKIT used to fit models, by hand:
- dot products of small vectors, and the shape of a data matrix (n rows, p columns)
- derivatives from the definition (limit / tiny nudge) for simple functions
- derivative rules: power, exp, log, sum, product, and especially the CHAIN RULE
- partial derivatives and the gradient of functions of 2-3 variables, evaluated at a point
- differentiating a sum of squared errors like g(z) = sum (z*x_i - y_i)^2 and setting it to zero to find the best z
- Pearson correlation by hand for 3-5 data points
Please:
1. Use plain, everyday language and explain any technical term in one short sentence.
2. Give me ONE question at a time, easy first, then harder. Use small whole numbers so I can work by hand.
3. Wait for my answer, then mark it and show the full working step by step, naming the rule used at each step.
4. If I make a mistake, explain what went wrong and give me a similar question.
5. Every 5 questions, summarise what I'm good at and what to practise.