ML3.5 · LESSON 5 OF 6

Precision, recall and F1

Compute logistic probabilities and evaluate the tradeoff introduced by a classification threshold.

THE IDEA

Accuracy can look perfect while the model is useless

Accuracy is the share of right answers. When positives are rare it rewards ignoring them. A fraud detector that never flags anything is right about 95.2% of the time on the data below, and finds 0 frauds.

positive rate4%

Model "never flag anything" on 1000 cases, 40 of them positive:

predicted positivepredicted notactually positiveactually not

accuracy 0.96precision 0recall 0F1 0flagged 0

Accuracy 96%, recall 0. It looks excellent while catching nothing.

A real model on 1500 transactions (72 fraud), threshold 0.5:

predicted positivepredicted notactually positiveactually not

accuracy 0.966precision 0.784recall 0.403F1 0.532flagged 37

Do-nothing accuracy is 95.2%. The real model scores 96.6% and also finds 29 frauds. Accuracy barely separates them. Recall (0.4 against 0) does.

Raise the positive rate on the left. A model that flags nothing keeps a high accuracy long after it has become worthless. The right side shows a real model on the fraud data.

Precision, recall and F1

These two ask different questions and have different denominators. Take 3 hits, 1 false alarm and 6 misses (TP = 3, FP = 1, FN = 6):

Metric 1 of 3

Precision = TP / (TP + FP)

TP = 3 over predicted positives = 3 + 1 = 4 = 0.75

Of everything I flagged, how much was real? Falls when you raise false alarms.

When a denominator is zero (no predicted positives, or no actual positives) the metric is undefined. Libraries make you choose a policy. Here we report 0.0 and say so, rather than letting a nan travel through the report.

Predict first. A model flags exactly one transaction and it is real fraud. There are 72 frauds in all. What are its precision and recall?

Choosing a threshold from costs

F1 treats a miss and a false alarm as equally bad. Real systems rarely do. If a missed fraud costs $40 and a false alarm costs $10, the cheapest threshold is where the total of both is lowest. Move the slider to find it:

threshold0.5

flagged 37caught 29missed 43false alarms 8

precision 0.78recall 0.4F1 0.532cost $1800

With these 1500 transactions, at threshold 0.5 the model flags 37, catches 29 of 72 frauds (recall 0.4, precision 0.78). The cost-optimal threshold when misses are expensive is far lower than 0.5, because paying for a few extra false alarms is cheaper than missing frauds.

Chapter context

A classifier often estimates a score or probability before a decision rule turns it into a label. The decision threshold is part of the application policy; it is not the same thing as the fitted probability model.

TRY IT YOURSELF

Part 1. Do it with the controls

All three thresholds start at 1.0: the model that flags nothing. 1500 transactions, 72 of them fraud. Scenario A and B use the same model but different costs, so the best threshold changes.

Scenario A: miss $40, false alarm $10

threshold A1

flagged 0caught 0missed 72false alarms 0

precision 0recall 0F1 0cost $2880

Scenario B: miss $10, false alarm $30

threshold B1

flagged 0caught 0missed 72false alarms 0

precision 0recall 0F1 0cost $720

No money: maximise F1

threshold C1

flagged 0caught 0missed 72false alarms 0

precision 0recall 0F1 0

To do: Scenario A (a miss costs $40, a false alarm $10): get the total cost within 10% of the cheapest threshold
To do: Scenario B (a miss costs $10, a false alarm $30): same goal, new costs
To do: Ignore money: find a threshold where F1 is at least 0.5

Part 2. Do it in code

precision_recall_f1(tp, fp, fn) returns (precision, recall, f1). When a denominator is 0, that metric is 0.0 (do not raise, do not return nan).
total_cost(y_true, y_pred, cost_fn, cost_fp) is the number of missed positives times cost_fn plus false alarms times cost_fp.
best_cost_threshold(y_true, proba, cost_fn, cost_fp) tries every distinct probability as a threshold (flag when proba >= threshold) and also float('inf') (flag nothing). Return the threshold with the lowest total cost; when several tie, return the largest.

import numpy as np

def precision_recall_f1(tp, fp, fn):
    # (precision, recall, f1); a 0 denominator means 0.0
    pass

def total_cost(y_true, y_pred, cost_fn, cost_fp):
    pass

def best_cost_threshold(y_true, proba, cost_fn, cost_fp):
    pass

print(precision_recall_f1(2, 1, 1))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

TP=2,FP=1,FN=1: find precision and recall.

APrecision 1/3 and recall 2/3, because FP and FN are swapped in the denominatorsBBoth are 2/3.CPrecision 2/4 and recall 2/4, because both divide by the total of all examples

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

\sigma(z)=\frac1{1+e^{-z}},\quad L=-y\log p-(1-y)\log(1-p).

P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN},\quad F1=\frac{2PR}{P+R}.

Define a policy for zero denominators rather than silently treating undefined metrics as universally meaningful.

Worked example

For actual labels [1,1,1,0,0] and predictions [1,1,0,1,0], TP=2, FP=1, FN=1 and TN=1. Accuracy is 3/5=0.6; precision, recall and F1 are all 2/3. Accuracy and F1 differ because they summarize different error patterns.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose, log
truth, pred = [1,1,1,0,0], [1,1,0,1,0]
tp = sum(a == b == 1 for a,b in zip(truth,pred))
fp = sum(a == 0 and b == 1 for a,b in zip(truth,pred))
fn = sum(a == 1 and b == 0 for a,b in zip(truth,pred))
precision, recall = tp/(tp+fp), tp/(tp+fn)
assert isclose(2*precision*recall/(precision+recall), 2/3)
assert sum(a == b for a,b in zip(truth,pred))/len(truth) == .6
assert isclose(-log(.8), .2231435513142097)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Accuracy can hide poor rare-class recall. A probability estimate does not choose the correct business threshold by itself. Ranking quality and calibration are separate properties.