M7.4 · LESSON 4 OF 6

Entropy

Connect likelihood, estimation and uncertainty to entropy, cross-entropy, KL divergence and perplexity.

THE IDEA

Surprise: rare things carry more information

If a friend tells you the sun rose today, you learn nothing. If they tell you a lottery ticket won, you learn a lot. The surprise (self-information) of an outcome with probability p is −log p: 0 for a certain event, and growing as the event gets rarer. With the base-2 logarithm the unit is the bit; with the natural logarithm it is the nat. An event of probability 1/8 carries 3 bits, because 8 = 2³: you need 3 yes/no questions to pin it down.

Entropy: the average surprise

Entropy is the expected surprise when outcomes really occur with probabilities p: H(p) = −Σ p·log p. Each term is an outcome's probability times its surprise.

Entropy of the distribution 1/2, 1/4, 1/8, 1/8 in bits 1 of 4

outcome A: p = 0.5, surprise = −log₂(0.5) = 1 bits

contribution p × surprise = 0.5

running total = 0.5 bits

weight A6weight B3weight C2weight D1

show nats instead of bits

	A	B	C	D
p	0.5	0.25	0.167	0.083
surprise (bits)	1	2	2.585	3.585
p × surprise	0.5	0.5	0.431	0.299
entropy	1.7296 bits

Reshape the distribution. Make one outcome certain, then make all four equal. Switch to nats to see the same quantity in a different unit.

Predict first. A fair coin versus a coin that lands heads 90 percent of the time. Which has higher entropy?

Edge cases and units

An outcome with probability 0 would have infinite surprise, but it never happens, and the term is p × surprise = 0 × infinity. The standard convention is 0 · log 0 = 0: impossible outcomes add nothing. A distribution with a single certain outcome therefore has entropy exactly 0.

Entropy in code: skip the zeros, and name the unit

import math
def entropy(p, base=2):
    return -sum(x * math.log(x, base) for x in p if x > 0)

entropy([0.5, 0.5])            # 1   bits
entropy([0.5, 0.5], math.e)    # 0.6931  nats  (= ln 2)
entropy([1.0, 0.0])            # 0.0   (not an error: 0 log 0 is skipped)

Chapter context

Statistics estimates properties of a population from observations. Information measures describe how surprising those observations are under a probability model. Keep the true or empirical distribution distinct from the model used to describe it.

TRY IT YOURSELF

Part 1. Tune it by hand

The four weights are normalised into probabilities. Tasks tick from the real entropy in bits.

weight A6weight B3weight C2weight D1

show nats instead of bits

	A	B	C	D
p	0.5	0.25	0.167	0.083
surprise (bits)	1	2	2.585	3.585
p × surprise	0.5	0.5	0.431	0.299
entropy	1.7296 bits

To do: Make one outcome certain (all the weight on a single outcome) so the entropy is exactly 0 bits
To do: Make the four outcomes equally likely, which gives the maximum possible entropy: 2 bits
To do: Shape the distribution so its entropy is 1.5 bits (within 0.01)
To do: Use exactly two outcomes with equal weights, the rest 0: a fair coin, exactly 1 bit

Part 2. Do it in code

Write entropy(p, base=2): minus the sum of p log p, in bits by default. Terms with p = 0 contribute 0 (the convention 0 log 0 = 0).
Raise ValueError if p has a negative entry or does not sum to 1 (tolerance 1e-9).
Write surprise(prob, base=2): the information of one outcome, -log(prob). A probability of 0 gives float("inf").

import math

def surprise(prob, base=2):
    # -log(prob) in the given base; infinite for probability 0
    pass

def entropy(p, base=2):
    # average surprise under p; skip zero probabilities; validate p
    pass

print(entropy([0.5, 0.5]))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

What is the entropy of a fair binary outcome in bits?

ATwo bits, because there are two outcomes.BOne bit.CZero bits, because a fair outcome carries no surprise.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

H(p)=-\sum_i p_i\log p_i,\quad H(p,q)=-\sum_i p_i\log q_i.

D_{KL}(p\Vert q)=H(p,q)-H(p),\quad \operatorname{PPL}=e^{H(p,q)}.

These expressions use natural logarithms. If q assigns zero probability where p is positive, cross-entropy and KL are infinite.

For n Bernoulli observations with s successes, the maximum-likelihood probability estimate is s/n. For a sample with mean x-bar, the variance estimate dividing squared deviations by n is a different quantity from the usual unbiased sample-variance estimator dividing by n-1. State which one is being used.

Worked example

Use p=[0.5,0.5] and q=[0.25,0.75]. Source entropy is ln 2≈0.693147 nats. Cross-entropy is −0.5ln(0.25)−0.5ln(0.75)≈0.836988. KL≈0.143841 nats. Perplexity is exp(0.836988)≈2.309401, larger than the source's effective uncertainty of 2.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log, exp, isclose
p, q = [.5,.5], [.25,.75]
h = -sum(a*log(a) for a in p)
ce = -sum(a*log(b) for a,b in zip(p,q))
assert isclose(h, 0.6931471805599453)
assert isclose(ce, 0.8369882167858358)
assert isclose(ce-h, 0.1438410362258905)
assert isclose(exp(ce), 2.309401076758503)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Likelihood is not the posterior probability of a model. Lower perplexity across different tokenizers is not a clean model comparison. Bias is not removed merely by computing more decimal places.