ML8.1 · LESSON 1 OF 6

Prior and likelihood

Update a prior, compare posterior means with MAP estimates and interpret predictive uncertainty.

THE IDEA

Two ingredients, and a coin to test them on

You find a coin and want its chance of landing heads, call it theta. Before flipping anything you may already believe something: most coins are close to fair, but a trick coin is possible. That belief, written as a distribution over theta, is the prior. We use a Beta distribution because it lives on the interval from 0 to 1 and has two knobs: Beta(1, 1) is flat (no opinion), Beta(2, 2) leans gently toward fair, and Beta(20, 20) is a firm belief that theta is near 0.5.

Then you flip. The likelihood answers a different question: for each possible theta, how probable is the data you saw? For h heads and t tails in a particular order, it is theta^h (1 - theta)^t. Seven heads and three tails is most probable when theta = 0.7, much less probable when theta = 0.2.

prior Beta(2, 2)likelihood (rescaled to fit)posterior Beta(9, 5)

prior a2prior b2heads7tails3

likelihood = theta^7 (1 - theta)^3 peaks at heads / flips = 7/10 = 0.7 area under the likelihood = B(8, 4) = 7.58e-4 area under prior x likelihood = 9.32e-4 (the evidence) posterior = (prior x likelihood) / 9.32e-4 posterior peak = 0.667

The likelihood curve does not have area 1, so it is not a distribution over theta.

Move the sliders. The orange likelihood is redrawn at a convenient height; its true area is printed on the right.

A likelihood is not a distribution over theta

The likelihood is a function of theta with the data held fixed. It tells you how well each theta explains the data, but nothing forces the curve to enclose an area of 1. For 7 heads and 3 tails its area is B(8, 4) = 1/1320 = 0.000758. Reading its height as "the probability that theta is 0.7" would be like reading a thermometer in the wrong units.

Predict first. You see 5 heads and 5 tails. Which of these is true of the likelihood curve theta^5 (1 - theta)^5?

Prior times likelihood, then normalise

Bayes' rule multiplies the two curves point by point: posterior(theta) is proportional to prior(theta) x likelihood(theta). The product still is not a distribution. Its area is the probability of the data averaged over the prior, called the evidence. For a Beta(2, 2) prior and 3 heads, 1 tail in one specific order the evidence is 0.05714 (that is 2/35). Divide the product by it and the area becomes 1: that is the posterior. You will watch it happen in the next lesson.

Chapter context

Probabilistic modeling represents uncertainty explicitly instead of presenting every fitted number as certain. Priors, likelihoods and approximation choices are assumptions that should be exposed rather than hidden behind a library call.

TRY IT YOURSELF

Part 1. Do it hands-on

Set the sliders. Tasks are checked from the exact numbers behind the plot and stay ticked once earned.

prior Beta(1, 1)likelihood (rescaled to fit)posterior Beta(1, 1)

prior a1prior b1heads0tails0

likelihood = theta^0 (1 - theta)^0 no flips yet: likelihood = 1 for every theta area under the likelihood = B(1, 1) = 1 area under prior x likelihood = 1 (the evidence) posterior = (prior x likelihood) / 1 posterior peak = 0

With no flips the likelihood is flat at 1 and has area 1, by coincidence.

To do: Choose data of exactly 8 flips whose likelihood curve peaks at 0.25
To do: Show that a likelihood is not a distribution: pick data with at least 10 flips where the area under the likelihood curve is below 0.001
To do: With data of 8 heads and 2 tails, choose a prior that pulls the posterior peak down to 0.55 or lower

Part 2. Do it in code

likelihood(theta, heads, tails): theta^heads * (1 - theta)^tails for a numpy array of candidate biases. Raise ValueError for negative counts.
log_likelihood(theta, heads, tails): the same on the log scale without producing NaN. A zero count contributes 0 even when theta is 0 or 1; a positive count at theta 0 (or 1) gives -inf.
unnormalized_posterior(theta, a, b, heads, tails): Beta(a, b) prior kernel theta^(a-1) (1-theta)^(b-1) times the likelihood. Raise ValueError unless a > 0 and b > 0.
normalize(theta, values): divide by the area under the curve using the trapezoid rule, np.sum((v[1:] + v[:-1]) / 2 * np.diff(theta)). Raise ValueError if the area is not a positive finite number.

import numpy as np

def likelihood(theta, heads, tails):
    pass

def log_likelihood(theta, heads, tails):
    pass

def unnormalized_posterior(theta, a, b, heads, tails):
    pass

def normalize(theta, values):
    pass

grid = np.linspace(0, 1, 5)
print(likelihood(grid, 3, 1))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Is a likelihood already a probability distribution over parameters?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For Bernoulli success probability theta and a Beta prior:

p(\theta\mid D)=\operatorname{Beta}(a+s,b+f),\quad E[\theta\mid D]=\frac{a+s}{a+b+s+f}.

For Beta(alpha,beta) with both parameters greater than one, the MAP is $(\alpha-1)/(\alpha+\beta-2)$ .

For a zero-mean Gaussian-process regression model with independent noise variance sigma-n squared, a test prediction has

\mu_*=k_*^T(K+\sigma_n^2I)^{-1}y,

v_*=k_{**}-k_*^T(K+\sigma_n^2I)^{-1}k_*.

K is the training covariance matrix, k-star connects the test input with training inputs, and k-star-star is its prior variance. This is latent-function variance; observation noise adds another term for a noisy predicted observation. Practical code solves systems instead of explicitly computing an inverse.

Worked example

Start with Beta(2,2), then observe three successes and one failure. The posterior is Beta(5,3). Its mean is 5/8=0.625. Its MAP is 4/6≈0.6667. Both summarize the same distribution but answer different questions.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
a, b = 2+3, 2+1
assert (a,b) == (5,3)
assert a/(a+b) == .625
assert isclose((a-1)/(a+b-2), 2/3)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A posterior estimate is conditional on the chosen model. MAP is not the same as a posterior mean. A small uncertainty under a misspecified model need not mean a reliable real-world answer.