ML3.4 · LESSON 4 OF 6

Confusion matrices

Compute logistic probabilities and evaluate the tradeoff introduced by a classification threshold.

THE IDEA

Four ways to be right or wrong

Once predictions are labels, every example falls in exactly one of four boxes. First declare which class is positive: usually the rare or important one (spam, fraud, disease). Then:

TP (true positive): actually positive, predicted positive. A hit.
FP (false positive): actually negative, predicted positive. A false alarm.
FN (false negative): actually positive, predicted negative. A miss.
TN (true negative): actually negative, predicted negative. A correct all-clear.

The first letter says whether the model was right (True or False), the second says what it predicted. Here is a spam filter on 12 emails, 1 means spam. Click a cell to see exactly which emails live there.

predicted 1predicted notactually 1actually not

False negatives: actually positive but predicted negative. The misses.

#1
y=1 ŷ=1

#2
y=0 ŷ=0

#3
y=1 ŷ=1

#4
y=0 ŷ=1

#5
y=0 ŷ=0

#6
y=1 ŷ=0

#7
y=1 ŷ=1

#8
y=0 ŷ=0

#9
y=1 ŷ=0

#10
y=0 ŷ=0

#11
y=0 ŷ=0

#12
y=1 ŷ=1

Positive class: 1. Highlighted examples belong to the selected cell.

Click each of the four cells and watch the highlighted emails. Rows are the truth, columns are the prediction.

The same data with the other class as positive

Nothing is built into the matrix that says 1 is the positive class. If you call class 0 positive, hits and correct rejections trade places, and so do misses and false alarms. The numbers in the table below are the same data viewed the other way:

predicted 0predicted notactually 0actually not

False negatives: actually positive but predicted negative. The misses.

#1
y=0 ŷ=0

#2
y=1 ŷ=1

#3
y=0 ŷ=0

#4
y=1 ŷ=0

#5
y=1 ŷ=1

#6
y=0 ŷ=1

#7
y=0 ŷ=0

#8
y=1 ŷ=1

#9
y=0 ŷ=1

#10
y=1 ŷ=1

#11
y=1 ŷ=1

#12
y=0 ŷ=0

Positive class: 0. Highlighted examples belong to the selected cell.

Positive class is now 0. Compare each cell with the previous figure.

Predict first. In the spam table, the filter predicted spam for an email that is really fine. Which cell?

Common mistakes

The usual mistake is reading the two letters backwards: "false negative" is a miss (the prediction was negative and wrong), not "a negative that was falsely detected". A second is forgetting to say which class is positive: a report saying "FP = 5" is meaningless without it. And check that the four counts add up to the number of examples. If they do not, an example was lost or counted twice.

Chapter context

A classifier often estimates a score or probability before a decision rule turns it into a label. The decision threshold is part of the application policy; it is not the same thing as the fitted probability model.

TRY IT YOURSELF

Part 1. Do it with the controls

Twelve emails, actual and predicted labels shown. Count the four cells by hand for each choice of positive class. The checks only accept the right numbers. Cells you have not filled count as wrong.

#	1	2	3	4	5	6	7	8	9	10	11	12
actual	1	0	1	0	0	1	1	0	1	0	0	1
predicted	1	0	1	1	0	0	1	0	0	0	0	1

Positive class = 1 (spam)

TPFPFNTN

Positive class = 0 (not spam)

TPFPFNTN

To do: Count the actual positives (y = 1): how many did the filter catch (TP) and how many did it miss (FN)?
To do: Count the actual negatives (y = 0): how many false alarms (FP) and how many correct rejections (TN)?
To do: Now treat class 0 as the positive class and fill in the second matrix

Part 2. Do it in code

confusion_counts(y_true, y_pred, positive=1) returns (tp, fp, fn, tn) as plain ints, where the class equal to positive is the positive class. Raise ValueError if the two inputs have different lengths.
confusion_matrix(y_true, y_pred) returns a 2 by 2 numpy array laid out as [[tn, fp], [fn, tp]], with class 1 positive (rows are actual, columns are predicted).

import numpy as np

def confusion_counts(y_true, y_pred, positive=1):
    # (tp, fp, fn, tn)
    pass

def confusion_matrix(y_true, y_pred):
    # [[tn, fp], [fn, tp]]
    pass

print(confusion_counts([1, 1, 1, 0, 0], [1, 1, 0, 1, 0]))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

A positive prediction for a truly negative example is which count?

AA true negative, because the example is actually negativeBA false negative, because the example was actually negativeCA false positive.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

\sigma(z)=\frac1{1+e^{-z}},\quad L=-y\log p-(1-y)\log(1-p).

P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN},\quad F1=\frac{2PR}{P+R}.

Define a policy for zero denominators rather than silently treating undefined metrics as universally meaningful.

Worked example

For actual labels [1,1,1,0,0] and predictions [1,1,0,1,0], TP=2, FP=1, FN=1 and TN=1. Accuracy is 3/5=0.6; precision, recall and F1 are all 2/3. Accuracy and F1 differ because they summarize different error patterns.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose, log
truth, pred = [1,1,1,0,0], [1,1,0,1,0]
tp = sum(a == b == 1 for a,b in zip(truth,pred))
fp = sum(a == 0 and b == 1 for a,b in zip(truth,pred))
fn = sum(a == 1 and b == 0 for a,b in zip(truth,pred))
precision, recall = tp/(tp+fp), tp/(tp+fn)
assert isclose(2*precision*recall/(precision+recall), 2/3)
assert sum(a == b for a,b in zip(truth,pred))/len(truth) == .6
assert isclose(-log(.8), .2231435513142097)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Accuracy can hide poor rare-class recall. A probability estimate does not choose the correct business threshold by itself. Ranking quality and calibration are separate properties.