Naive Bayes
Like LDA, but with one bold shortcut: pretend that, within each class, the inputs have nothing to do with each other. Then you can handle each input on its own and multiply.
Question type: classification.
Week: 3.
Book: ISLP 4.4.4.
Techniques on this page: the independence assumption, per-input pictures (bell curves for numbers, counts for categories), multiplying and applying Bayes’ theorem.
The problem it solves
LDA needs to describe how all the inputs behave together in each class: not just each input’s spread, but how every pair moves together. With many inputs that’s a lot to estimate, and with little data the estimates are shaky (high variance).
Naive Bayes skips the hard part. It assumes that within a class, knowing one input tells you nothing about another. Then you only need a simple one-input description per input per class, which needs far less data. It’s “naive” because that’s rarely exactly true, but it often works well anyway, especially with many inputs and not much data (spam filters are the classic example).
Words you’ll hear
| Word | Plain meaning |
|---|---|
| independent | knowing one thing doesn’t change the chances of the other |
| \(f_{kj}(x_j)\) | how typical the value \(x_j\) of input \(j\) is within class \(k\) |
| \(\prod_j\) | multiply over all the inputs |
| prior, posterior | as in LDA: chance before and after seeing the inputs |
| underflow | multiplying many tiny numbers gives a number so small the computer rounds it to 0 |
Technique 1: the independence shortcut
Problem it solves: describing all the inputs together is hard; describing them one at a time is easy.
\[f_k(x) = f_{k1}(x_1)\times f_{k2}(x_2)\times\cdots\times f_{kp}(x_p)\]
“How typical is this whole customer in class \(k\)” becomes “how typical is their balance in class \(k\)” times “how typical is their student status in class \(k\)”, and so on.
Technique 2: one picture per input
Problem it solves: different inputs are different kinds of thing, so each can get its own kind of description.
- Numeric input (balance): a bell curve per class, with that class’s own average and spread for that input.
- Category input (student yes/no): just count. The chance of “student = yes” in class \(k\) is the fraction of class \(k\) training points who are students.
Technique 3: multiply, then Bayes’ theorem
- For each class, look up each input’s value in that class’s picture.
- Multiply them, and multiply by the prior \(\pi_k\).
- Divide each class’s result by the total so they add to 1:
\[P(Y = k \mid x) = \frac{\pi_k\,f_{k1}(x_1)\cdots f_{kp}(x_p)}{\sum_l \pi_l\,f_{l1}(x_1)\cdots f_{lp}(x_p)}\]
- Pick the biggest.
In code, step 2 is done by adding logs instead of multiplying, to avoid underflow.
Worked example: spam filter. \(P(\text{spam}) = 0.4\). “free” appears in 50% of spam and 5% of normal emails. “meeting” appears in 10% of spam and 40% of normal emails. An email contains both words.
- Spam: \(0.4 \times 0.5 \times 0.1 = 0.020\)
- Normal: \(0.6 \times 0.05 \times 0.4 = 0.012\)
- \(P(\text{spam} \mid \text{email}) = 0.020/(0.020 + 0.012) = 0.625\). Flag it as spam.
Worked example: balance and student. Balance in £1000s.
| prior | balance average | balance spread (sd) | P(student) | |
|---|---|---|---|---|
| no default | 0.7 | 0.8 | 0.4 | 0.3 |
| default | 0.3 | 1.8 | 0.4 | 0.5 |
New customer: balance 1.4, student.
- Balance bell-curve heights: no default, \((1.4 - 0.8)/0.4 = 1.5\) sds from centre, height \(0.324\). Default, \(-1\) sd, height \(0.605\).
- Student: \(0.3\) and \(0.5\).
- Multiply with priors: no default \(0.7 \times 0.324 \times 0.3 = 0.068\); default \(0.3 \times 0.605 \times 0.5 = 0.091\).
- \(P(\text{default}) = 0.091/(0.068 + 0.091) = 0.57\). Predict default.
Naive Bayes vs LDA
| LDA | Naive Bayes | |
|---|---|---|
| inputs allowed to move together within a class? | yes | no (that’s the shortcut) |
| spread | same for every class | each class can have its own |
| mixes numbers and categories easily? | not really | yes |
| best when | moderate number of inputs, bell-shaped data | many inputs, little data |
Why the shortcut often helps: it adds some bias (the assumption is wrong) but removes a lot of variance (far fewer things to estimate). The same trade-off as everywhere in this course.
What to read in the book
| Section | Read? | Why |
|---|---|---|
| 4.4.4 Naive Bayes | Read | Core, including the worked example in Table 4.9 |
| 4.5.1 Analytical comparison | Skim | Where naive Bayes fits relative to LDA and logistic regression |
Practice prompt
I'm studying CS5100 Data Analysis (textbook ISLP, section 4.4.4). I want practice questions on NAIVE BAYES, done by hand:
- what the independence-within-class assumption means and why it's "naive"
- spam-filter style questions with word probabilities per class and a prior
- mixed inputs: a categorical predictor (estimate by counting) and a numeric predictor (Gaussian with a per-class mean and sd; give me the density values or the z-scores so I can work by hand)
- multiplying the per-predictor probabilities with the prior and normalising to get posteriors
- estimating the per-class probabilities from a small training table
- why naive Bayes can work well when p is large and n small (bias vs variance)
- how naive Bayes differs from LDA
- why code adds logs instead of multiplying
Please:
1. Use plain, everyday language and explain any technical term in one short sentence.
2. Give me ONE question at a time, easy first, then harder, with friendly numbers.
3. Wait for my answer, then mark it and show the full working step by step.
4. If I get it wrong, explain the mistake and give me a similar question.
5. Every 5 questions, summarise my strengths and what to practise.