The Bayes classifier
The "perfect" classifier: always pick the most likely category. It's the gold standard every other classifier tries to copy.
Question type: classification (the answer is a category).
Week: 1 (and it comes back as the final step of every week 3 model).
Book: ISLP 2.2.3, first half.
Techniques on this page: conditional probability, argmax, decision boundary, Bayes error rate.
The problem it solves
When the answer is a category (default or not, spam or not), what is the best possible rule for guessing? Before building real models, it helps to know what “best” means, so we can tell how close a real model gets. The Bayes classifier answers that: if you somehow knew the true chances of each category, the best you can do is pick the most likely one.
The catch: in real life we never know those true chances. So this isn’t a model you can use directly. It’s the target. Every classifier in week 3 (logistic regression, LDA, naive Bayes) and KNN in week 1 is really just a different way of estimating those chances and then doing what the Bayes classifier does.
The idea in everyday words
A weather forecaster says “70% chance of rain tomorrow.” If you have to bet on rain or no rain, you bet rain. You’ll still be wrong 30% of the time, but no other bet does better. That’s the Bayes classifier.
Words you’ll hear
| Word | Plain meaning |
|---|---|
| \(P(Y = k \mid X = x_0)\) | “the chance the answer is category \(k\), given that the information is \(x_0\)”. The bar \(\mid\) means “given”. |
| conditional probability | a chance worked out for a specific situation, like “chance of default for someone with a £2000 balance” |
| posterior probability | the same thing; the chance after seeing the information |
| \(\arg\max_k\) | “the \(k\) that gives the biggest value”. Not the biggest value itself, but which one it is |
| decision boundary | the line on a chart where the classifier switches from one category to another |
| Bayes error rate | how often even the perfect classifier is wrong. A floor no model can go below |
Technique 1: conditional probability and the argmax rule
Problem it solves: turning “chances” into a single decision.
\[\hat Y_{\text{Bayes}}(x_0) = \arg\max_k\ P(Y = k \mid X = x_0)\]
In words: for this new case, list the chance of each category, and pick the category with the biggest chance.
With two categories (0 and 1), the chances add up to 1, so this becomes simply:
\[\text{predict } 1 \text{ if } P(Y = 1 \mid X = x_0) > 0.5, \text{ otherwise predict } 0.\]
Worked examples.
- Two categories, \(P(Y=1 \mid x_0) = 0.7\). Predict 1.
- Three categories with chances \((0.2,\ 0.5,\ 0.3)\). The biggest is 0.5, so predict category 2 (the second one). Here “max” is 0.5 and “argmax” is category 2.
- Two categories, \(P(Y=1\mid x_0) = 0.45\). Since \(0.45 < 0.5\), predict 0.
Technique 2: the decision boundary
Problem it solves: seeing the rule as a picture.
If you have two inputs \(X_1\) and \(X_2\), imagine colouring every point on a chart by what the classifier would predict there. The line where the colour switches is the decision boundary. For the Bayes classifier with two categories, it’s exactly where the chance is a 50/50 tie:
\[P(Y = 1 \mid X_1, X_2) = 0.5\]
In the slides (and ISLP Figure 2.13) it’s the dashed purple line between the orange and blue regions. It can be any shape, curvy or straight, depending on the true chances.
Technique 3: the Bayes error rate
Problem it solves: “how good can any classifier possibly be on this problem?”
At a single point \(x_0\), if the best category has chance \(\max_k P(Y=k\mid x_0)\), then the chance of being wrong there is
\[1 - \max_k P(Y = k \mid x_0)\]
Average that over all the points you might see and you get the Bayes error rate. It’s not zero whenever categories overlap (some people with the same balance default and some don’t). It’s the classification version of the “luck” \(\epsilon\) from the big picture: the part nobody can predict.
Worked examples.
- \(P(Y=1 \mid x_0) = 0.7\). Chance of being wrong at \(x_0\): \(1 - 0.7 = 0.3\).
- Chances \((0.2, 0.5, 0.3)\): wrong with chance \(1 - 0.5 = 0.5\), even when playing perfectly.
- Suppose \(X\) can only be \(a\) (60% of the time) or \(b\) (40%). At \(a\), \(P(Y=1) = 0.9\); at \(b\), \(P(Y=1) = 0.3\).
- At \(a\): predict 1, wrong with chance \(0.1\).
- At \(b\): predict 0 (since \(0.3 < 0.5\)), wrong with chance \(0.3\).
- Overall Bayes error rate: \(0.6 \times 0.1 + 0.4 \times 0.3 = 0.06 + 0.12 = 0.18\).
How it connects to other models
- KNN (page) estimates the chances by looking at nearby training points, then applies this rule.
- Logistic regression (page) estimates the chances with an S-shaped curve, then applies this rule.
- LDA and naive Bayes (LDA, naive Bayes) estimate the chances using Bayes’ theorem, then apply this rule.
- In week 3 the “0.5” becomes adjustable (a threshold), when some mistakes cost more than others. See judging classifiers.
Careful: the Bayes classifier (this page, “pick the most likely”) and Bayes’ theorem (week 3, a formula for flipping chances around) share a name but are different things.
What to read in the book
| Section | Read? | Why |
|---|---|---|
| ISLP 2.2.3 “The Classification Setting”, up to “K-Nearest Neighbors” | Read | Error rate, Bayes classifier, Bayes error rate |
| Figure 2.13 | Study | The picture of the Bayes decision boundary used on the slides |
| The rest of 2.2.3 | see KNN |
Practice prompt
I'm studying CS5100 Data Analysis (textbook ISLP, section 2.2.3). I want practice questions on the BAYES CLASSIFIER:
- reading conditional probabilities P(Y = k | X = x0) and choosing the class with argmax
- the difference between max and argmax
- the two-class rule "predict 1 if P(Y=1|x0) > 0.5"
- what the Bayes decision boundary is and where it lies
- computing the error probability 1 - max_k P(Y=k|x0) at a point
- computing an overall Bayes error rate when X takes a few values with given probabilities
- why the Bayes classifier is a gold standard but can't be used directly in practice
Please:
1. Use plain, everyday language and explain any technical term in one short sentence.
2. Give me ONE question at a time, easy first, then harder. Use small, friendly numbers.
3. Wait for my answer, then mark it and show the full working step by step.
4. If I get it wrong, explain the misunderstanding and give me a similar question.
5. Include a few "explain in your own words" questions, not just calculations.
6. Every 5 questions, summarise what I'm solid on and what needs work.