Linear discriminant analysis (LDA)
Instead of asking "given this balance, will they default?", LDA first describes what balances look like in each group, then flips that around with Bayes' theorem.
Question type: classification.
Week: 3.
Book: ISLP 4.4 intro, 4.4.1, 4.4.2 (skim the maths).
Techniques on this page: Bayes’ theorem, priors, densities, the bell curve (Gaussian), estimating means and a pooled spread, the discriminant score, the decision boundary.
The problem it solves
Logistic regression models the chance of each class directly. LDA takes the opposite route, called a generative approach:
- Describe each group separately: “defaulters tend to have balances around £1800, non-defaulters around £800, both spread out by about the same amount.”
- Use Bayes’ theorem to turn that into “given this balance, what’s the chance they’re a defaulter?”
Why go the long way round? Describing each group is often easier and more stable, especially with little data or when the groups are well separated (logistic regression’s coefficients can run off to infinity in that case). And it handles several classes naturally. The price is an assumption: that each group’s values form a bell curve.
Words you’ll hear
| Word | Plain meaning |
|---|---|
| generative model | describes how the inputs look within each class, then flips it |
| prior, \(\pi_k\) (“pi k”) | how common class \(k\) is overall, before looking at any input. Nothing to do with the number 3.14 here |
| \(f_k(x)\) | how typical the value \(x\) is within class \(k\) |
| posterior, \(p_k(x)\) | the chance of class \(k\) after seeing \(x\). What we want |
| density | a curve whose height says how common values near \(x\) are (for measurements, where exact values have zero chance) |
| Gaussian, normal distribution | the bell curve |
| \(\mu_k\) (“mu k”) | the centre (average) of class \(k\) |
| \(\sigma^2\) (“sigma squared”), variance | how spread out the values are. LDA assumes every class has the same spread |
| discriminant, \(\delta_k(x)\) (“delta k”) | a score for each class; the biggest score wins |
Technique 1: Bayes’ theorem (flipping the question)
Problem it solves: we know “how common is a high balance among defaulters?” but we want “how likely is a default among high-balance people?” Those are different questions.
The easiest way to see it is by counting. 1000 customers, 30 default. 80% of defaulters have a high balance; 10% of non-defaulters do.
- Defaulters with a high balance: \(30 \times 0.8 = 24\).
- Non-defaulters with a high balance: \(970 \times 0.1 = 97\).
- So among the 121 high-balance customers, 24 default: \(24/121 = 0.198\).
That calculation is Bayes’ theorem. In symbols:
\[p_k(x) = P(Y = k \mid X = x) = \frac{\pi_k\,f_k(x)}{\sum_l \pi_l\,f_l(x)}\]
Top: (how common the class is) × (how typical \(x\) is in it). Bottom: the same for every class, added up, so the answers sum to 1.
Lesson: even though a high balance is 8 times more typical of defaulters, the chance is only about 20%, because defaulters are rare. Priors matter.
Don’t confuse Bayes’ theorem (this flipping formula) with the Bayes classifier (week 1, “pick the most likely class”). LDA uses both: the theorem to get the chances, then the classifier to choose.
Technique 2: the bell curve (assuming a shape for each class)
Problem it solves: \(f_k(x)\) could be any shape, and estimating an arbitrary shape needs lots of data. LDA assumes a bell curve, which is described by just a centre and a spread:
\[f_k(x) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\Big(-\frac{(x - \mu_k)^2}{2\sigma^2}\Big)\]
Read it in pieces: \((x - \mu_k)^2\) is how far \(x\) is from the class centre; dividing by \(2\sigma^2\) scales that by the spread; \(\exp(-\dots)\) turns “far” into “small”, giving the bell with its peak at \(\mu_k\); the bit in front just makes the total area 1.
The key assumption: same spread \(\sigma^2\) for every class, different centres \(\mu_k\).
Technique 3: estimating the pieces from data
Problem it solves: the formula needs \(\pi_k\), \(\mu_k\) and \(\sigma^2\). Fitting LDA is just counting and averaging. No gradient descent needed.
\[\hat\pi_k = \frac{n_k}{n}, \qquad \hat\mu_k = \text{average of } x \text{ in class } k, \qquad \hat\sigma^2 = \frac{1}{n - m}\sum_k\ \sum_{i \in \text{class } k}(x_i - \hat\mu_k)^2\]
\(n_k\) is how many training points are in class \(k\), and \(m\) is the number of classes. The spread is pooled: each point is measured from its own class’s centre, then everything is added up.
Slide error: equation (31) measures the spread from the overall average instead of each class’s own average. That’s much too big when the classes are far apart. Use the book’s version (eq. 4.20), shown above.
Technique 4: the discriminant score (a shortcut for comparing classes)
Problem it solves: you don’t need the full posterior to pick a winner. The bottom of Bayes’ theorem is the same for every class, so just compare the tops. Taking logs (never changes which is biggest) and dropping the pieces that are the same for every class leaves:
\[\delta_k(x) = \frac{x\,\mu_k}{\sigma^2} - \frac{\mu_k^2}{2\sigma^2} + \log\pi_k \qquad\text{predict the class with the biggest } \delta_k(x)\]
This is a straight line in \(x\), which is where the name linear discriminant analysis comes from. Deriving it needs only algebra and log rules.
Technique 5: the decision boundary
With two classes, set \(\delta_0(x) = \delta_1(x)\) and solve:
\[x^* = \frac{\mu_0 + \mu_1}{2} + \frac{\sigma^2}{\mu_1 - \mu_0}\log\frac{\pi_0}{\pi_1}\]
With equal priors the log is 0 and the boundary is halfway between the centres. Unequal priors nudge it towards the rarer class, giving the common class more room.
Full worked example. Class 0: \(x = (1, 2, 3)\). Class 1: \(x = (5, 6, 7, 8)\). So \(n = 7\), \(m = 2\).
- Priors: \(\hat\pi_0 = 3/7 = 0.429\), \(\hat\pi_1 = 4/7 = 0.571\).
- Centres: \(\hat\mu_0 = 2\), \(\hat\mu_1 = 6.5\).
- Pooled spread: class 0 squared gaps \(1 + 0 + 1 = 2\); class 1: \(2.25 + 0.25 + 0.25 + 2.25 = 5\). \(\hat\sigma^2 = 7/(7 - 2) = 1.4\).
- Classify \(x = 4\):
- \(\delta_0(4) = \frac{4 \times 2}{1.4} - \frac{4}{2.8} + \log\frac37 = 5.714 - 1.429 - 0.847 = 3.438\)
- \(\delta_1(4) = \frac{4 \times 6.5}{1.4} - \frac{42.25}{2.8} + \log\frac47 = 18.571 - 15.089 - 0.560 = 2.923\)
- Class 0 has the bigger score: predict class 0.
- Boundary: \(x^* = 4.25 + \frac{1.4}{4.5}\log\frac{3}{4} = 4.25 - 0.09 = 4.16\). Slightly left of the midpoint because class 1 is more common; \(x = 4\) is just on the class 0 side.
Several inputs (skim)
With \(p\) inputs, each class becomes a bell-shaped “hill” in \(p\) dimensions with its own centre \(\mu_k\) (a list of averages) and a shared shape \(\Sigma\) (“big sigma”, the table of spreads and how inputs move together). The score becomes \(\delta_k(x) = x^T\Sigma^{-1}\mu_k - \tfrac12\mu_k^T\Sigma^{-1}\mu_k + \log\pi_k\), still a straight-line boundary. You only need the idea, not the calculation.
What to read in the book
| Section | Read? | Why |
|---|---|---|
| 4.4 intro (generative models) | Read | Bayes’ theorem, priors, densities |
| 4.4.1 LDA for one predictor | Read | Core. Use its eq. 4.20 for the spread, not slide eq. 31 |
| 4.4.2 LDA for more than one predictor | Skim the maths | Then read carefully from the confusion matrix to the end: see judging classifiers |
| 4.4.3 QDA | Skip | Not on the slides (it’s LDA with a different spread per class) |
| 4.5.1 Analytical comparison | Skim | Useful for “which method when” |
Practice prompt
I'm studying CS5100 Data Analysis (textbook ISLP, sections 4.4-4.4.2). I want practice questions on LINEAR DISCRIMINANT ANALYSIS (LDA), done by hand:
- Bayes' theorem as counting (e.g. 1000 people, a % with a condition, test results), then with the formula pi_k f_k(x) / sum
- why priors matter (rare classes)
- the difference between Bayes' theorem and the Bayes classifier
- estimating priors, class means and the POOLED within-class variance (divide by n - m) from small 1D data sets
- computing discriminant scores delta_k(x) = x*mu_k/sigma^2 - mu_k^2/(2 sigma^2) + log(pi_k) and choosing the class
- finding the two-class decision boundary, with equal and unequal priors
- why LDA is called linear, and when it might beat logistic regression
- reading the Gaussian density formula piece by piece
Please:
1. Use plain, everyday language and explain any technical term in one short sentence.
2. Give me ONE question at a time, easy first, then harder. Use small data sets (2-4 points per class) with friendly numbers.
3. Wait for my answer, then mark it and show the full working step by step.
4. If I get it wrong, explain the mistake and give me a similar question.
5. Every 5 questions, summarise my strengths and what to practise.