ML3.6 · LESSON 6 OF 6

ROC, PR and calibration

Compute logistic probabilities and evaluate the tradeoff introduced by a classification threshold.

THE IDEA

Sweep the threshold and trace a curve

Every threshold gives one (false-positive rate, true-positive rate) pair. Sweep t from high to low and the points trace the ROC curve. A model with no skill lies on the diagonal. A perfect one hugs the top-left corner. The area under it, the AUC, equals the chance that a random positive is scored above a random negative. The precision-recall curve traces precision against recall the same way.

threshold0.5

ROC AUC 0.931average precision 0.882FPR 0.129TPR (recall) 0.779precision 0.757

Dashed lines are a model with no skill: the diagonal for ROC, the positive rate (34%) for precision-recall. Switch to the rare-positive data: ROC stays looking healthy while precision collapses at high recall.

Move the threshold and watch the dot on both curves. Then switch to the data where only about 5% are positive.

ROC rates divide by the class sizes, so a flood of negatives barely moves the false-positive rate. Precision divides by what you flagged, and there the many negatives hurt. With rare positives the ROC curve can look excellent while the precision-recall curve shows how many of your flags are wrong. Use PR when positives are rare and you care about the flagged list.

Predict first. On the rare-positive data the ROC AUC looks high but the average precision is much lower. Is something wrong with the calculation?

Calibration: do the probabilities mean what they say?

A model is calibrated if, among examples it gives about 70%, about 70% really are positive. The reliability diagram bins the predictions and plots observed frequency against average predicted probability. A calibrated model follows the diagonal. Below it means over-confident. The model below ranks examples well (AUC 0.885) but its logits are stretched 2.5 times too far. Dividing the logits by a temperature T shrinks them. Find the T that puts the dots on the diagonal.

temperature T1

ECE 0.116mean predicted 0.261really positive 0.305ROC AUC 0.885

Dots below the diagonal mean the model claims more than happens. Changing T squeezes or stretches the probabilities, but the AUC (a ranking property) does not move at all.

ECE is the weighted average gap between the dots and the diagonal. Notice the ROC AUC never changes with T.

Chapter context

A classifier often estimates a score or probability before a decision rule turns it into a label. The decision threshold is part of the application policy; it is not the same thing as the fitted probability model.

TRY IT YOURSELF

Part 1. Do it with the controls

400 examples, 122 positive. The green box on the ROC plot is the operating region you need. Both T and the threshold are yours to set.

temperature T1threshold0.9

ECE 0.116mean predicted 0.261really positive 0.305ROC AUC 0.885FPR 0.032TPR 0.393

Dots below the diagonal mean the model claims more than happens. Changing T squeezes or stretches the probabilities, but the AUC (a ranking property) does not move at all.

To do: Rescale the over-confident model with the temperature T until the calibration error (ECE) is 0.065 or lower
To do: Make the average predicted probability match the real positive rate to within 0.02
To do: Pick an operating point on the ROC curve with false-positive rate at most 0.1 and true-positive rate at least 0.6

Part 2. Do it in code

roc_points(y_true, scores) returns two arrays (fpr, tpr). Start at (0, 0), then add one point for each distinct score from the highest down, flagging every example with score >= that value. Examples with the same score move together. Raise ValueError if either class is missing.
auc(fpr, tpr) is the area under that curve by the trapezoid rule.
expected_calibration_error(y_true, proba, n_bins=10) splits [0, 1] into equal bins (p = 1.0 belongs in the last bin) and returns the weighted average of |observed positive rate - mean predicted probability| over the non-empty bins. Empty input gives 0.0.

import numpy as np

def roc_points(y_true, scores):
    # return (fpr, tpr) as numpy arrays, starting at (0, 0)
    pass

def auc(fpr, tpr):
    # trapezoid rule
    pass

def expected_calibration_error(y_true, proba, n_bins=10):
    pass

print(roc_points([0, 0, 1, 1], [0.1, 0.4, 0.35, 0.8]))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does a high ranking AUC guarantee calibrated probabilities?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

\sigma(z)=\frac1{1+e^{-z}},\quad L=-y\log p-(1-y)\log(1-p).

P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN},\quad F1=\frac{2PR}{P+R}.

Define a policy for zero denominators rather than silently treating undefined metrics as universally meaningful.

Worked example

For actual labels [1,1,1,0,0] and predictions [1,1,0,1,0], TP=2, FP=1, FN=1 and TN=1. Accuracy is 3/5=0.6; precision, recall and F1 are all 2/3. Accuracy and F1 differ because they summarize different error patterns.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose, log
truth, pred = [1,1,1,0,0], [1,1,0,1,0]
tp = sum(a == b == 1 for a,b in zip(truth,pred))
fp = sum(a == 0 and b == 1 for a,b in zip(truth,pred))
fn = sum(a == 1 and b == 0 for a,b in zip(truth,pred))
precision, recall = tp/(tp+fp), tp/(tp+fn)
assert isclose(2*precision*recall/(precision+recall), 2/3)
assert sum(a == b for a,b in zip(truth,pred))/len(truth) == .6
assert isclose(-log(.8), .2231435513142097)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Accuracy can hide poor rare-class recall. A probability estimate does not choose the correct business threshold by itself. Ranking quality and calibration are separate properties.