Judging classifiers: confusion matrix, ROC and AUC
Not a model. These are the tools for scoring any classifier, and for choosing where to draw the line between "yes" and "no".
Used with: every classifier: logistic regression, LDA, naive Bayes, KNN.
Week: 3. The slides stop before this, but the lab uses it (ROC curves and AUC).
Book: ISLP 4.4.2, from the confusion matrix (Table 4.4) to the end of the section.
Techniques on this page: the confusion matrix and its rates, moving the threshold, the ROC curve, AUC.
The problem it solves
The error rate treats every mistake the same. But in the Default data only about 3% of customers default. A “model” that always says no is 97% accurate and catches zero defaulters. Useless, but it looks great.
Two things are missing: (1) a breakdown of which mistakes you make, and (2) a way to judge a model across every possible threshold, not just 0.5. That’s what this page gives you. It’s also why the lab balances the data 50/50 before training.
Words you’ll hear
| Word | Plain meaning |
|---|---|
| positive / negative | the two classes. Positive is usually the one you’re hunting for (defaults) |
| true positive (TP) | said yes, really yes |
| false positive (FP) | said yes, really no (a false alarm) |
| true negative (TN) | said no, really no |
| false negative (FN) | said no, really yes (a miss) |
| sensitivity, recall, TPR | of the real positives, the fraction you caught |
| specificity, TNR | of the real negatives, the fraction you correctly cleared |
| FPR | of the real negatives, the fraction you wrongly flagged. $= 1 - $ specificity |
| threshold, \(\xi\) (“xi”) | the cut-off: predict yes if the chance is at least \(\xi\) |
| ROC curve | a chart of catch-rate against false-alarm-rate for every threshold |
| AUC | area under the ROC curve: one score from 0.5 (guessing) to 1 (perfect) |
Technique 1: the confusion matrix
Problem it solves: shows the four kinds of outcome instead of one blended number.
| predicted no | predicted yes | total | |
|---|---|---|---|
| really no | TN | FP | N |
| really yes | FN | TP | P |
From it:
\[\text{TPR} = \frac{TP}{P},\quad \text{TNR} = \frac{TN}{N},\quad \text{FPR} = \frac{FP}{N},\quad \text{accuracy} = \frac{TP + TN}{P + N}\]
TP = 40, FN = 10, FP = 30, TN = 120. So P = 50, N = 150. TPR \(= 40/50 = 0.8\). TNR \(= 120/150 = 0.8\). FPR \(= 0.2\). Accuracy \(= 160/200 = 0.8\).
Technique 2: moving the threshold
Problem it solves: sometimes one kind of mistake is worse. A bank would rather wrongly flag a few good customers than miss defaulters.
\[\hat y = 1 \text{ if } \hat p(x) \ge \xi, \text{ otherwise } 0\]
The Bayes classifier uses \(\xi = 0.5\). Lower \(\xi\) (say 0.2) and you catch more positives (TPR up) but raise more false alarms (FPR up). There’s always a trade-off.
Technique 3: the ROC curve
Problem it solves: compares models without committing to one threshold.
Plot TPR (up) against FPR (across) for every threshold:
- Threshold above every score: nothing predicted yes, point \((0, 0)\).
- Threshold 0: everything predicted yes, point \((1, 1)\).
- A perfect model reaches the top-left corner \((0, 1)\): catch everyone, no false alarms.
- A useless model (random scores) follows the diagonal.
The lab draws these with sklearn.metrics.roc_curve.
Technique 4: AUC (one number for the whole curve)
Problem it solves: “which ROC curve is better?” in a single score.
AUC is the area under the curve: 1 is perfect, 0.5 is random guessing. It has a lovely plain meaning: the chance that a randomly picked positive gets a higher score than a randomly picked negative. That gives a way to compute it by hand: count pairs.
Worked example. Eight test points, sorted by predicted chance:
| \(\hat p\) | 0.90 | 0.80 | 0.70 | 0.60 | 0.55 | 0.40 | 0.30 | 0.20 |
|---|---|---|---|---|---|---|---|---|
| \(y\) | 1 | 1 | 0 | 1 | 0 | 1 | 0 | 0 |
ROC points:
- \(\xi = 0.75\): top two predicted yes. TP = 2, FP = 0. TPR = 0.5, FPR = 0.
- \(\xi = 0.5\): top five. TP = 3, FP = 2. TPR = 0.75, FPR = 0.5.
- \(\xi = 0.35\): top six. TP = 4, FP = 2. TPR = 1, FPR = 0.5.
AUC by counting pairs. Positives score 0.9, 0.8, 0.6, 0.4. Negatives score 0.7, 0.55, 0.3, 0.2. 0.9 beats all 4 negatives; 0.8 beats 4; 0.6 beats 3; 0.4 beats 2. Total 13 out of \(4 \times 4 = 16\) pairs: AUC = 0.8125.
Why the lab balances the classes
With 3% defaulters, a model can score a low error rate by almost always saying “no”, and gradient descent is dominated by the “no” rows. The lab keeps every defaulter and randomly picks the same number of non-defaulters, so the data is 50/50 and an error rate of 0.5 means “no better than guessing”.
What to read in the book
| Section | Read? | Why |
|---|---|---|
| 4.4.2, from Table 4.4 to the end | Read carefully | Confusion matrix, threshold 0.2 (Table 4.5), the ROC curve (Fig. 4.8), the terms (Tables 4.6, 4.7) |
| Earlier part of 4.4.2 | Skim | Covered on the LDA page |
Practice prompt
I'm studying CS5100 Data Analysis (textbook ISLP, section 4.4.2). I want practice questions on EVALUATING CLASSIFIERS:
- filling in a confusion matrix from a list of true labels and predictions
- computing TPR (sensitivity), TNR (specificity), FPR, FNR, accuracy and error rate
- why accuracy misleads with imbalanced classes, with a numeric example
- applying different thresholds to a list of predicted probabilities and computing TPR/FPR at each
- sketching or describing ROC points, and what the corners and diagonal mean
- computing AUC by hand by counting (positive, negative) pairs
- choosing a threshold when false negatives cost more than false positives
- why the lab balances the classes by subsampling
Please:
1. Use plain, everyday language and explain any technical term in one short sentence.
2. Give me ONE question at a time, easy first, then harder, using 6-10 test points.
3. Wait for my answer, then mark it and show the full working step by step.
4. If I get it wrong, explain the mistake and give me a similar question.
5. Every 5 questions, summarise my strengths and what to practise.