ML8.3 · LESSON 3 OF 6

MAP and posterior summaries

Update a prior, compare posterior means with MAP estimates and interpret predictive uncertainty.

THE IDEA

One distribution, several one-number answers

The posterior is a whole curve. People often want a single number or a range from it. The common summaries answer different questions:

MAP (maximum a posteriori): the single most probable value, the peak. For Beta(a, b) with a, b > 1 it is (a - 1) / (a + b - 2).
Posterior mean: the probability-weighted average, a / (a + b). It is the best guess if you are penalised by squared error.
Credible interval: a range holding, say, 95% of the posterior probability. The equal-tailed version leaves 2.5% in each tail.

For the worked example Beta(5, 3): mean = 5/8 = 0.625, MAP = 4/6 = 0.6667, median = 0.6359, and the 95% interval is [0.2904, 0.901]. Each is correct for its own question.

posterior Beta(2, 5)mean = 0.286median = 0.264MAP = 0.2

prior a1prior b1heads1tails4

posterior = Beta(1 + 1, 1 + 4) = Beta(2, 5) mean = 2/7 = 0.286 MAP = (2 - 1)/(2 + 5 - 2) = 0.2 median = 0.264 95% interval = [0.043, 0.641] (2.5% of the mass in each tail) sd = 0.16

Mean minus MAP = 0.086.

Use the presets, then change the sliders. When the curve is lopsided the three markers separate; when it is symmetric they sit together.

When they disagree

With a flat prior and 0 heads in 4 flips the posterior is Beta(1, 5). Its peak is at theta = 0: the MAP says the coin can never land heads. The mean is 1/6 = 0.167, which keeps some room for heads. This is why a single number can mislead and why intervals matter: the 95% interval here is [0.005, 0.522].

Predict first. For Beta(2, 5), the MAP is 1/5 = 0.2. Is the mean larger, smaller or equal?

MAP depends on how you parametrise

The mean of a function of theta is not generally the function of the mean, but the MAP is even more fragile: it changes if you describe the same belief in different units. Take the odds phi = theta / (1 - theta). For Beta(5, 3) the MAP of theta is 2/3, which is odds 2. But if you maximise the density of the odds itself (a beta-prime curve, with mode (a - 1)/(b + 1)) the peak is at 1, which corresponds to theta = 0.5. Same posterior, two different MAPs, because stretching the axis changes where the density is tallest.

Chapter context

Probabilistic modeling represents uncertainty explicitly instead of presenting every fitted number as certain. Priors, likelihoods and approximation choices are assumptions that should be exposed rather than hidden behind a library call.

TRY IT YOURSELF

Part 1. Do it hands-on

The posterior is Beta(prior a + heads, prior b + tails). Tasks use the exact summaries shown in the panel.

posterior Beta(1, 1)mean = 0.5median = 0.5

prior a1prior b1heads0tails0

posterior = Beta(1 + 0, 1 + 0) = Beta(1, 1) mean = 1/2 = 0.5 MAP = undefined (flat or U-shaped: no single peak) median = 0.5 95% interval = [0.025, 0.975] (2.5% of the mass in each tail) sd = 0.289

To do: Observe exactly 6 flips and make the MAP equal the posterior mean (and not a boundary)
To do: Find a posterior with an interior MAP (both parameters above 1) where the mean is more than 0.08 above the MAP
To do: Make the 95% credible interval narrower than 0.2 while the MAP stays between 0.4 and 0.6

Part 2. Do it in code

map_estimate(a, b): the mode of Beta(a, b). If a > 1 and b > 1 it is (a - 1) / (a + b - 2). If a <= 1 < b it is 0.0, and if b <= 1 < a it is 1.0. If both are at most 1 there is no single mode: raise ValueError. Also raise ValueError unless a > 0 and b > 0.
posterior_mean(a, b): a / (a + b).
credible_interval(a, b, level=0.95): the equal-tailed interval (lower, upper), computed with numpy only: evaluate the density kernel on a midpoint grid of 100000 points in log space (so large a and b do not underflow), normalize, take the cumulative sum and read the quantiles with np.interp. Raise ValueError unless 0 < level < 1.

import numpy as np

def map_estimate(a, b):
    pass

def posterior_mean(a, b):
    pass

def credible_interval(a, b, level=0.95):
    pass

print(map_estimate(5, 3), posterior_mean(5, 3))   # 0.666..., 0.625

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Are posterior mean and MAP always equal?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For Bernoulli success probability theta and a Beta prior:

p(\theta\mid D)=\operatorname{Beta}(a+s,b+f),\quad E[\theta\mid D]=\frac{a+s}{a+b+s+f}.

For Beta(alpha,beta) with both parameters greater than one, the MAP is $(\alpha-1)/(\alpha+\beta-2)$ .

For a zero-mean Gaussian-process regression model with independent noise variance sigma-n squared, a test prediction has

\mu_*=k_*^T(K+\sigma_n^2I)^{-1}y,

v_*=k_{**}-k_*^T(K+\sigma_n^2I)^{-1}k_*.

K is the training covariance matrix, k-star connects the test input with training inputs, and k-star-star is its prior variance. This is latent-function variance; observation noise adds another term for a noisy predicted observation. Practical code solves systems instead of explicitly computing an inverse.

Worked example

Start with Beta(2,2), then observe three successes and one failure. The posterior is Beta(5,3). Its mean is 5/8=0.625. Its MAP is 4/6≈0.6667. Both summarize the same distribution but answer different questions.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
a, b = 2+3, 2+1
assert (a,b) == (5,3)
assert a/(a+b) == .625
assert isclose((a-1)/(a+b-2), 2/3)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A posterior estimate is conditional on the chosen model. MAP is not the same as a posterior mean. A small uncertainty under a misspecified model need not mean a reliable real-world answer.