ML3.2 · LESSON 2 OF 6

Binary cross-entropy

Compute logistic probabilities and evaluate the tradeoff introduced by a classification threshold.

THE IDEA

A loss that punishes confident mistakes

How should we score a probability? If the truth is 1 and the model said 0.8, the model gave the right class an 80% chance. The binary cross-entropy (log loss) charges −ln(0.8) = 0.2231. If the truth is 0, we use the probability given to class 0, which is 1 − p. In one formula: L = −y·ln(p) − (1 − y)·ln(1 − p). Only one of the two terms is ever active.

predicted p0.8

true class	loss at p = 0.8
1	0.2231
0	1.6094

The curve for class 1 is flat near p = 1 and shoots up as p approaches 0. Being confidently wrong is not twice as bad as being unsure. It is dramatically worse.

Drag p. The orange curve is the cost when the truth is 1, the blue curve when the truth is 0.

p given to the true class	0.9	0.6	0.5	0.2	0.05	0.01
loss = −ln p	0.105	0.511	0.693	1.609	2.996	4.605

Going from 0.9 to 0.5 costs 0.59 more. Going from 0.05 to 0.01 costs 1.61 more even though both are "wrong". The curve is steep near zero, so training is pushed hard to avoid confident errors. That is also why this loss, unlike accuracy, has a gradient to follow even when the label is already predicted correctly: it keeps rewarding more confidence in the right class.

The dataset loss is an average of contributions

Training uses the mean over all examples. A table of contributions shows where the loss comes from. A few confident mistakes can dominate everything else.

A (y=1)0.9B (y=1)0.8C (y=1)0.3D (y=0)0.1E (y=0)0.7F (y=1)0.05

ex	y	p	p for true class	loss
A	1	0.9	0.9	0.105
B	1	0.8	0.8	0.223
C	1	0.3	0.3	1.204
D	0	0.1	0.9	0.105
E	0	0.7	0.3	1.204
F	1	0.05	0.05	2.996
mean loss				0.973

Largest single contribution: example F at 2.996, which is 51% of the total. p = 0 or 1 is clipped to 1e-12, so the worst possible cost is 27.63 instead of infinity.

Six examples with true classes. Slide any probability and see its row and the total change. The red row is the biggest contributor.

Predict first. Model A is 99% sure on every example and is wrong on 1 of 100. Model B says 0.7 on every example and is right on all 100. Which has the lower mean loss?

Common mistakes: log of zero, and training from probabilities

If a model outputs exactly p = 0 for an example whose label is 1, the loss is −ln 0 = infinity and everything downstream becomes inf or nan. Implementations clip p into [ε, 1 − ε] (here ε = 1e-12, so the worst loss is 27.63). For actual training, work from the logit z, not from p: max(z, 0) − z·y + ln(1 + e^(−|z|)) never builds an overflowing number and is exact even at z = ±1000. You will write both in the practice.

Chapter context

A classifier often estimates a score or probability before a decision rule turns it into a label. The decision threshold is part of the application policy; it is not the same thing as the fitted probability model.

TRY IT YOURSELF

Part 1. Do it with the controls

Six examples with fixed true classes (y). You choose the probability for each. The red row is the worst contributor right now.

A (y=1)0.9B (y=1)0.8C (y=1)0.3D (y=0)0.1E (y=0)0.7F (y=1)0.05

ex	y	p	p for true class	loss
A	1	0.9	0.9	0.105
B	1	0.8	0.8	0.223
C	1	0.3	0.3	1.204
D	0	0.1	0.9	0.105
E	0	0.7	0.3	1.204
F	1	0.05	0.05	2.996
mean loss				0.973

Largest single contribution: example F at 2.996, which is 51% of the total. p = 0 or 1 is clipped to 1e-12, so the worst possible cost is 27.63 instead of infinity.

To do: Bring the average loss down to 0.30 or lower
To do: Make sure no single example pays more than 0.6
To do: See the explosion once: make some example pay more than 4 (give it a probability under 0.02 for its true class), then fix it again

Part 2. Do it in code

bce(y, p, eps=1e-12) returns the mean binary cross-entropy. Clip p into [eps, 1 - eps] first so p = 0 or p = 1 never produces infinity. An empty input has loss 0.0.
bce_with_logits(y, z) returns the same mean loss but straight from the raw scores z, without ever forming a probability. It must stay finite and exact for z = 1000 or z = -1000.

import numpy as np

def bce(y, p, eps=1e-12):
    # mean of -(y*log(p) + (1-y)*log(1-p)), with p clipped to [eps, 1-eps]
    pass

def bce_with_logits(y, z):
    # same loss from logits: max(z, 0) - z*y + log(1 + exp(-|z|)), averaged
    pass

print(bce([1], [0.8]))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

If y=1 and predicted probability is .8, what is the loss?

AAbout 0.2, because the loss equals the probability left over for the other classB-ln(.8), approximately .2231.CAbout 0.8, because the model gave the true class an 80 percent chance

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

\sigma(z)=\frac1{1+e^{-z}},\quad L=-y\log p-(1-y)\log(1-p).

P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN},\quad F1=\frac{2PR}{P+R}.

Define a policy for zero denominators rather than silently treating undefined metrics as universally meaningful.

Worked example

For actual labels [1,1,1,0,0] and predictions [1,1,0,1,0], TP=2, FP=1, FN=1 and TN=1. Accuracy is 3/5=0.6; precision, recall and F1 are all 2/3. Accuracy and F1 differ because they summarize different error patterns.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose, log
truth, pred = [1,1,1,0,0], [1,1,0,1,0]
tp = sum(a == b == 1 for a,b in zip(truth,pred))
fp = sum(a == 0 and b == 1 for a,b in zip(truth,pred))
fn = sum(a == 1 and b == 0 for a,b in zip(truth,pred))
precision, recall = tp/(tp+fp), tp/(tp+fn)
assert isclose(2*precision*recall/(precision+recall), 2/3)
assert sum(a == b for a,b in zip(truth,pred))/len(truth) == .6
assert isclose(-log(.8), .2231435513142097)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Accuracy can hide poor rare-class recall. A probability estimate does not choose the correct business threshold by itself. Ranking quality and calibration are separate properties.