Judging classifiers
A single error rate hides which mistakes a classifier makes. The confusion matrix counts them, the threshold trades one kind for the other, and the ROC curve and AUC summarise every threshold at once. Then: which of the week's four classifiers to use when.
The week 3 lab ends with ROC curves and AUC for logistic regression and KNN. The slides stop at KNN, so this stage follows ISLP 4.4.2 and the lab.
Book: what to read for this stage
ISLP 4.4.2 from the confusion matrix (Table 4.4) to the end of the section: thresholds, sensitivity and specificity, the ROC curve and AUC (Fig. 4.8, Tables 4.6 and 4.7). Read it. 4.5.1 analytical comparison: skim the conclusions, skip the algebra. 4.5.2: skip.
1. KNN as a classifier, and the four-way comparison
KNN classification (stage 4) estimates \(\hat p_k(x)\) as the fraction of the \(K\) nearest training points in class \(k\); with two classes, \(\hat p_1(x)\) is just the mean of the neighbours’ 0/1 labels (the lab’s np.mean(Ny)). It makes no assumption about the classes’ shape and can draw curved boundaries, but needs lots of data, is slow to predict, needs normalised inputs, and its \(K\) must be chosen on validation data. With \(K = 4\) and a 2-2 vote, np.argmax picks class 0: odd \(K\) avoids ties.
| Method | How it estimates \(P(Y = k \mid x)\) | Boundary | Watch out for |
|---|---|---|---|
| Logistic regression | directly, sigmoid (or softmax) of a linear function | straight | separable data: coefficients grow forever without a penalty |
| LDA | Gaussian per class with shared variance, then Bayes | straight | relies on the Gaussian assumption; good with small \(n\) when it holds |
| Naive Bayes | independent inputs per class, then Bayes | can curve | the independence assumption; great when \(p\) is large |
| KNN | vote among the \(K\) nearest | any shape | needs lots of data, scaling, choice of \(K\) |
Training data \((x, y)\): \((1, 0), (2, 0), (3, 1), (4, 0), (5, 1), (6, 1)\). KNN with \(K = 5\) at \(x = 3.4\): what is \(\hat p_1\)?
Nearest five: \(x = 3\) (1), 4 (0), 2 (0), 5 (1), 1 (0). Two 1s out of five: \(\hat p_1 = 0.4\), predict 0.
2. The confusion matrix
| Predicted 0 | Predicted 1 | Total | |
|---|---|---|---|
| Actually 0 | TN | FP | N |
| Actually 1 | FN | TP | P |
- TPR, sensitivity, recall \(= TP/P\): of the real positives, how many did we catch?
- TNR, specificity \(= TN/N\): of the real negatives, how many did we correctly clear?
- FPR $= FP/N = 1 - $ specificity. FNR \(= FN/P\).
- Accuracy \(= (TP + TN)/(P + N)\); error rate $= 1 - $ accuracy.
Why accuracy misleads. Only 3.33% of the Default customers default. A “classifier” that always says no is 96.7% accurate and catches nobody (TPR 0). That’s why the lab subsamples the negatives to get a balanced data set before training.
3. Thresholds and the ROC curve
Predict 1 when \(\hat p(x) \ge \xi\). Lowering \(\xi\) from 0.5 catches more positives (TPR up) but flags more negatives wrongly (FPR up). A bank that badly wants to catch defaulters might use \(\xi = 0.2\).
The ROC curve plots TPR against FPR for every threshold. \(\xi\) above every score gives \((0, 0)\); \(\xi = 0\) gives \((1, 1)\). A perfect classifier reaches the top-left corner \((0, 1)\); random scores follow the diagonal.
The AUC (area under the curve) is 1 for perfect, 0.5 for random guessing. It equals the probability that a randomly chosen positive gets a higher score than a randomly chosen negative, which gives a way to compute it by counting pairs.
Worked example. Eight test points sorted by score: \((0.9, 1), (0.8, 1), (0.7, 0), (0.6, 1), (0.55, 0), (0.4, 1), (0.3, 0), (0.2, 0)\). \(P = N = 4\).
- \(\xi = 0.75\): top two predicted 1: TP 2, FP 0, so (FPR, TPR) \(= (0, 0.5)\).
- \(\xi = 0.5\): top five: TP 3, FP 2, so \((0.5, 0.75)\); accuracy \(5/8\).
- AUC by pairs: positive 0.9 beats all 4 negatives, 0.8 beats 4, 0.6 beats 3 (0.55, 0.3, 0.2), 0.4 beats 2. Total \(13/16 = 0.8125\).
Stage test
A classifier’s scores are random numbers unrelated to the labels. Roughly what is its AUC?
Random scores: a positive outscores a negative half the time. The AUC doesn’t depend on any single threshold.
Lowering the threshold \(\xi\) from 0.5 to 0.2 will:
More points are predicted positive, which catches more true positives and more false positives.
(3 marks.) In a data set where 3% of customers default, a model has 97% accuracy. Explain why this might be a poor model, and name a better way to judge it.
With 3% defaulters, the trivial rule “nobody defaults” is already 97% accurate while catching no defaulters at all, so 97% accuracy may mean the model is useless for the job. We should look at the confusion matrix, in particular the true positive rate (sensitivity: the fraction of real defaulters caught) and the false positive rate, possibly at a lower threshold, and summarise performance across all thresholds with the ROC curve and its AUC (0.5 is guessing, 1 is perfect). Balancing the classes for training, as the lab does, also helps.
Which classifier can draw a curved decision boundary with no assumption about the shape of the classes?
Logistic regression and LDA give straight boundaries (unless you add polynomial features).
TP = 40, FN = 10, FP = 30, TN = 120. What is the specificity (TNR)?
\(N = 150\), \(TNR = 120/150 = 0.8\).