ML3.3 · LESSON 3 OF 6

Thresholds and decisions

Compute logistic probabilities and evaluate the tradeoff introduced by a classification threshold.

THE IDEA

The model estimates. You decide.

The fitted model produces a number for each example. A decision rule then turns it into an action: "flag it if p ≥ t". The threshold t is a policy choice, not part of the model. You can change t without refitting anything, because the probabilities are untouched. Below are 200 examples scored by a model, 68 of them truly positive.

Blue: examples that are really negative. Orange: really positive. Everything to the right of the line is called positive.

threshold0.5

predicted positivepredicted notactually positiveactually not

accuracy 0.84precision 0.757recall 0.779F1 0.768flagged 70

The 200 probabilities never change. Only the line moves, and with it the labels, the counts and every metric.

Move the threshold. The probabilities (the bars) never move. The labels, counts and metrics all do.

Every threshold is a trade

Lowering t catches more real positives but also flags more negatives. Raising t does the opposite. For these 200 examples:

t	flagged	caught (TP)	false alarms (FP)	missed (FN)
0.3	95	62	33	6
0.5	70	53	17	15
0.7	48	43	5	25

Predict first. You lower the threshold from 0.5 to 0.3. What must be true about the count of false negatives?

The common mistake: 0.5 is not a law

0.5 is the default only because it is the point where the model thinks both classes are equally likely. A screening tool for a serious disease might use 0.1, because missing a case is much worse than a false alarm. An automatic account ban might use 0.95. And if you retrain with a different class balance, the best threshold can move. Choose t from the costs and capacity of your situation, which the next lessons quantify.

Chapter context

A classifier often estimates a score or probability before a decision rule turns it into a label. The decision threshold is part of the application policy; it is not the same thing as the fitted probability model.

TRY IT YOURSELF

Part 1. Do it with the controls

The same 200 scored examples. There is no single best threshold: each task wants a different one, so you need to move the slider several times. Capacity limit for the third task: 60 flagged.

Blue: examples that are really negative. Orange: really positive. Everything to the right of the line is called positive.

threshold0.5

predicted positivepredicted notactually positiveactually not

accuracy 0.84precision 0.757recall 0.779F1 0.768flagged 70

The 200 probabilities never change. Only the line moves, and with it the labels, the counts and every metric.

To do: Find a threshold that catches at least 95% of the real positives (recall 0.95 or more)
To do: Find a threshold where at least 90% of the flagged examples really are positive (precision 0.90 or more)
To do: A reviewer can check only 60 cases. Flag at most 60 and still keep recall within 0.03 of the best possible for that limit

Part 2. Do it in code

predict_labels(proba, threshold=0.5) returns an integer array: 1 where proba >= threshold, else 0. It must not change the probabilities you pass in.
threshold_for_recall(y_true, proba, min_recall) returns the largest threshold (one of the values in proba) whose recall is at least min_recall. Return None when there are no actual positives or the recall cannot be reached.

import numpy as np

def predict_labels(proba, threshold=0.5):
    # 1 where proba >= threshold else 0 (an integer array)
    pass

def threshold_for_recall(y_true, proba, min_recall):
    # try the distinct probabilities from high to low; return the first threshold
    # whose recall >= min_recall. None if there are no positives or it is unreachable.
    pass

p = np.array([0.9, 0.8, 0.6, 0.3, 0.55])
print(predict_labels(p, 0.5), predict_labels(p, 0.3))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does changing .5 to .3 alter the already computed probabilities?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

\sigma(z)=\frac1{1+e^{-z}},\quad L=-y\log p-(1-y)\log(1-p).

P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN},\quad F1=\frac{2PR}{P+R}.

Define a policy for zero denominators rather than silently treating undefined metrics as universally meaningful.

Worked example

For actual labels [1,1,1,0,0] and predictions [1,1,0,1,0], TP=2, FP=1, FN=1 and TN=1. Accuracy is 3/5=0.6; precision, recall and F1 are all 2/3. Accuracy and F1 differ because they summarize different error patterns.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose, log
truth, pred = [1,1,1,0,0], [1,1,0,1,0]
tp = sum(a == b == 1 for a,b in zip(truth,pred))
fp = sum(a == 0 and b == 1 for a,b in zip(truth,pred))
fn = sum(a == 1 and b == 0 for a,b in zip(truth,pred))
precision, recall = tp/(tp+fp), tp/(tp+fn)
assert isclose(2*precision*recall/(precision+recall), 2/3)
assert sum(a == b for a,b in zip(truth,pred))/len(truth) == .6
assert isclose(-log(.8), .2231435513142097)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Accuracy can hide poor rare-class recall. A probability estimate does not choose the correct business threshold by itself. Ranking quality and calibration are separate properties.