M7.2 · LESSON 2 OF 6

Likelihood

Connect likelihood, estimation and uncertainty to entropy, cross-entropy, KL divergence and perplexity.

THE IDEA

Fix the data, vary the parameter

A coin lands heads with unknown probability θ. You flip it 20 times and see 14 heads. Which θ is most believable? Probability normally asks: given θ, how likely is this data? Likelihood reads the same formula the other way round: the data is fixed, and we compare parameter values.

For one particular sequence with h heads and N − h tails, flips are independent, so the probabilities multiply: L(θ) = θ^h · (1 − θ)^(N−h). The maximum-likelihood estimate (MLE) is the θ that makes the observed data least surprising.

likelihood (scaled to its peak) log-likelihood (scaled)

heads in 2012 of 20theta, percent0.5

data: 12 heads, 8 tails

L(theta) = theta^12 (1 - theta)^8

L(0.5) = 9.54e-7

log L = 12 ln(0.5) + 8 ln(0.5) = -13.863

peak at heads / flips = 12/20 = 0.6

Change the number of heads and slide θ. The green line is the peak, the orange line is your θ. The peak sits at heads divided by flips.

Predict first. You see 7 heads in 20 flips. Where is the peak of the likelihood curve?

Why we take logs

Multiplying hundreds of probabilities, each below 1, gives a number so small that a float cannot hold it, and it silently becomes exactly 0. Taking the logarithm turns the product into a sum: log L = h·ln θ + (N − h)·ln(1 − θ). Because the log is increasing, the peak is at the same θ, and sums of moderate numbers are safe. For 14 heads in 20 flips at θ = 0.7, log L = -12.217.

flips N100

70 heads in 100 flips, theta = 0.7

product of 100 probabilities = 2.95e-27

sum of 100 log probabilities = -61.09

Drag N upward. The naive product hits exactly 0 and carries no information. The log version just keeps getting more negative.

With 70 percent heads, the product in this demo first becomes exactly 0 at about 1230 flips, which is a tiny data set by machine-learning standards.

Common mistakes

Likelihood is not the probability of the parameter. The curve does not add up to 1 over θ, and a high value does not mean "probably the truth". The estimate can be extreme. With 20 heads in 20 flips the MLE is 1.0: the model claims tails are impossible. Any later tail then has probability 0 and log-likelihood of minus infinity.

The same arithmetic in Python

import math
data = [1]*14 + [0]*6          # 14 heads, 6 tails
sum(math.log(0.7 if x else 0.3) for x in data)   # -12.2173
math.log(0.0)                    # ValueError: math domain error

Chapter context

Statistics estimates properties of a population from observations. Information measures describe how surprising those observations are under a probability model. Keep the true or empirical distribution distinct from the model used to describe it.

TRY IT YOURSELF

Part 1. Tune it by hand

Move the data and the parameter. The tasks tick from the state of both sliders.

likelihood (scaled to its peak) log-likelihood (scaled)

heads in 2012 of 20theta, percent0.5

data: 12 heads, 8 tails

L(theta) = theta^12 (1 - theta)^8

L(0.5) = 9.54e-7

log L = 12 ln(0.5) + 8 ln(0.5) = -13.863

peak at heads / flips = 12/20 = 0.6

To do: Keep the data at 12 heads out of 20 and slide the parameter to the value with the highest likelihood
To do: Change the data to 7 heads out of 20 and put the parameter at the new maximum-likelihood estimate
To do: Make the data all heads (20 of 20) and push the parameter as high as the slider goes. Notice the likelihood is still rising

Part 2. Do it in code

Write log_likelihood(data, theta) for a list of 0/1 coin flips (1 = heads): the sum of log probabilities of each flip. Use natural logs.
Treat 0 * log(0) as 0, so theta = 0 with all tails gives 0, but theta = 0 with any heads gives minus infinity (float("-inf")).
Write mle(data), the heads fraction, and naive_likelihood(data, theta), the plain product, to compare with the log version on a long sequence.
theta outside [0, 1] or data other than 0 and 1 must raise ValueError.

import math

def log_likelihood(data, theta):
    # sum over flips of log P(flip). Remember 0 * log(0) = 0
    pass

def mle(data):
    # the value of theta that maximises the likelihood
    pass

def naive_likelihood(data, theta):
    # multiply the probabilities of the flips one by one (no logs)
    pass

print(log_likelihood([1, 0, 1], 0.5))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Which model has higher likelihood for an observed event: probability .2 or .8?

AThe .8 model, for that observation alone.BThe .2 model, because it leaves more probability for other events.CBoth models have the same likelihood, since the data is fixed.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

H(p)=-\sum_i p_i\log p_i,\quad H(p,q)=-\sum_i p_i\log q_i.

D_{KL}(p\Vert q)=H(p,q)-H(p),\quad \operatorname{PPL}=e^{H(p,q)}.

These expressions use natural logarithms. If q assigns zero probability where p is positive, cross-entropy and KL are infinite.

For n Bernoulli observations with s successes, the maximum-likelihood probability estimate is s/n. For a sample with mean x-bar, the variance estimate dividing squared deviations by n is a different quantity from the usual unbiased sample-variance estimator dividing by n-1. State which one is being used.

Worked example

Use p=[0.5,0.5] and q=[0.25,0.75]. Source entropy is ln 2≈0.693147 nats. Cross-entropy is −0.5ln(0.25)−0.5ln(0.75)≈0.836988. KL≈0.143841 nats. Perplexity is exp(0.836988)≈2.309401, larger than the source's effective uncertainty of 2.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log, exp, isclose
p, q = [.5,.5], [.25,.75]
h = -sum(a*log(a) for a in p)
ce = -sum(a*log(b) for a,b in zip(p,q))
assert isclose(h, 0.6931471805599453)
assert isclose(ce, 0.8369882167858358)
assert isclose(ce-h, 0.1438410362258905)
assert isclose(exp(ce), 2.309401076758503)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Likelihood is not the posterior probability of a model. Lower perplexity across different tokenizers is not a clean model comparison. Bias is not removed merely by computing more decimal places.