M7.6 · LESSON 6 OF 6

Perplexity

Connect likelihood, estimation and uncertainty to entropy, cross-entropy, KL divergence and perplexity.

THE IDEA

From a loss to a count of choices

A language model reads a text and, at each position, gives a probability to the word that actually came next. The average negative log of these probabilities is the model's cross-entropy loss on that text, say L nats per token. Perplexity is exp(L). It converts the loss into a number you can picture: the size of a set of equally likely choices that would be equally hard.

If a model assigns probability 1/4 to every true next word, the loss is ln 4 and the perplexity is exactly 4: it behaves like someone choosing among 4 equally likely words each time. Perplexity 1 means certainty; perplexity 20 means the model is about as uncertain as a 20-way coin.

context	true next	p given	-ln p
the	cat	0.2	1.609
the cat	sat	0.2	1.609
the cat sat	on	0.2	1.609
the cat sat on	the	0.2	1.609
the cat sat on the	mat	0.2	1.609
mean loss (nats)			1.6094

perplexity = exp(mean loss) = exp(1.6094) = 5

about as unsure as a choice among 5 equally likely words

p(cat)0.2p(sat)0.2p(on)0.2p(the)0.2p(mat)0.2

Each slider is the probability the model gave to the word that actually came next, in steps of 0.05.

Set the probability the model gave to each true next word. Watch the mean loss and the perplexity. Try making four of five words perfect and one terrible.

Predict first. A model gives probabilities 0.5, 0.25, 0.1, 0.6 and 0.3 to the five true next words. What is its perplexity?

Units must match, and comparisons need care

Use the same base on both sides: e^(loss in nats) or 2^(loss in bits). A loss of ln 4 nats is 2 bits, and both give perplexity 4. Mixing them (for example exp of a loss measured in bits) gives a wrong number.

Perplexity from per-token probabilities

import math
probs = [0.5, 0.25, 0.1, 0.6, 0.3]
loss = sum(-math.log(p) for p in probs) / len(probs)   # nats
math.exp(loss)                                          # 3.385
bits = sum(-math.log2(p) for p in probs) / len(probs)
2 ** bits                                               # 3.385  (same)

Chapter context

Statistics estimates properties of a population from observations. Information measures describe how surprising those observations are under a probability model. Keep the true or empirical distribution distinct from the model used to describe it.

TRY IT YOURSELF

Part 1. Tune it by hand

The five sliders are the probabilities the model gave to the true next words. Tasks tick from the computed perplexity.

context	true next	p given	-ln p
the	cat	0.2	1.609
the cat	sat	0.2	1.609
the cat sat	on	0.2	1.609
the cat sat on	the	0.2	1.609
the cat sat on the	mat	0.2	1.609
mean loss (nats)			1.6094

perplexity = exp(mean loss) = exp(1.6094) = 5

about as unsure as a choice among 5 equally likely words

p(cat)0.2p(sat)0.2p(on)0.2p(the)0.2p(mat)0.2

Each slider is the probability the model gave to the word that actually came next, in steps of 0.05.

To do: Get the perplexity down to 4 or lower: as good as picking among four equally likely words
To do: Land the perplexity on 2 (within 0.02): the model is as unsure as a fair coin on every word
To do: Push the perplexity below 2 while one word still gets a probability of 0.10 or less

Part 2. Do it in code

Write mean_nll(probs, base=math.e): the mean of -log(p) over the probabilities a model gave to the tokens that actually occurred.
Write perplexity(probs, base=math.e): base raised to mean_nll computed with the same base. The answer must not depend on the base you choose.
A probability of 0 gives float("inf"); an empty list or a probability outside [0, 1] raises ValueError.
Write perplexity_from_loss(loss, base=math.e) that turns an average loss into a perplexity.

import math

def mean_nll(probs, base=math.e):
    # average of -log(p) in the given base
    pass

def perplexity(probs, base=math.e):
    # base ** mean_nll(probs, base)
    pass

def perplexity_from_loss(loss, base=math.e):
    # loss is an average negative log probability measured in the given base
    pass

print(perplexity([0.25, 0.25, 0.25]))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

What is perplexity when mean natural-log loss is ln 4?

ATwo, because ln 4 is converted to a base-2 logarithm.BAbout 1.39, the loss itself.CFour.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

H(p)=-\sum_i p_i\log p_i,\quad H(p,q)=-\sum_i p_i\log q_i.

D_{KL}(p\Vert q)=H(p,q)-H(p),\quad \operatorname{PPL}=e^{H(p,q)}.

These expressions use natural logarithms. If q assigns zero probability where p is positive, cross-entropy and KL are infinite.

For n Bernoulli observations with s successes, the maximum-likelihood probability estimate is s/n. For a sample with mean x-bar, the variance estimate dividing squared deviations by n is a different quantity from the usual unbiased sample-variance estimator dividing by n-1. State which one is being used.

Worked example

Use p=[0.5,0.5] and q=[0.25,0.75]. Source entropy is ln 2≈0.693147 nats. Cross-entropy is −0.5ln(0.25)−0.5ln(0.75)≈0.836988. KL≈0.143841 nats. Perplexity is exp(0.836988)≈2.309401, larger than the source's effective uncertainty of 2.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log, exp, isclose
p, q = [.5,.5], [.25,.75]
h = -sum(a*log(a) for a in p)
ce = -sum(a*log(b) for a,b in zip(p,q))
assert isclose(h, 0.6931471805599453)
assert isclose(ce, 0.8369882167858358)
assert isclose(ce-h, 0.1438410362258905)
assert isclose(exp(ce), 2.309401076758503)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Likelihood is not the posterior probability of a model. Lower perplexity across different tokenizers is not a clean model comparison. Bias is not removed merely by computing more decimal places.