M6.4 · LESSON 4 OF 6

Expectation and variance

Calculate conditional probabilities, Bayes updates, expectations, variance and softmax probabilities.

THE IDEA

Expectation is a probability-weighted average

The expected value of a random quantity X is each possible value times its probability, added up: E[X] = sum of x P(X = x). For a fair die that is (1 + 2 + 3 + 4 + 5 + 6) / 6 = 3.5. Note that 3.5 is not a possible roll. The expectation is the balance point of the distribution, not a prediction for one roll.

Predict first. A loaded die gives 1 to 5 probability 0.1 each and 6 probability 0.5. What is E[X]?

Averages converge, and how fast depends on the spread

Simulate the game and keep a running average. Early on it jumps around; as draws pile up it settles on E[X]. The green band is the typical size of the wobble, about 2 standard deviations over the square root of the number of draws. Four times as many draws only halves it. Change the game and the seed to see that convergence is a statistical tendency, not a fixed path.

Game:Draws: 100

blue: average of the first k draws. green dashed: E[X] = 3.5. Green band: E[X] plus or minus 2 sd/sqrt(k).

after 100 draws: average = 3.51, gap from E[X] = 0.01

spread: sample variance = 3.09, exact Var(X) = 2.917, sd = 1.708

value	1	2	3	4	5	6
exact P	0.167	0.167	0.167	0.167	0.167	0.167
seen	0.16	0.2	0.16	0.12	0.17	0.19

Pick a game and the number of draws. The sample variance should land near the exact variance in the formula line.

Variance measures spread

Two games can have the same mean and feel completely different. Var(X) = E[(X - E[X])^2]: the average squared distance from the mean. The calm game (3 or 4) and the wild game (1 or 6) both have E[X] = 3.5, but variances 0.25 and 6.25. The standard deviation is the square root, which has the same units as X. Switch the game above and compare the width of the green band and the bars.

weight of 11weight of 21weight of 31weight of 41weight of 51weight of 61

probability of a face = its weight / 6

E[X] = sum of face x probability = 3.5

Var(X) = average of (face - 3.5)^2 = 2.917, sd = 1.708

Set integer weights for the six faces. Probabilities are weight / total. Watch E[X] and Var(X) respond.

Predict first. Which has the larger variance: always rolling 3 or 4 (equal chance), or always rolling 1 or 6 (equal chance)?

Common mistakes

The mean of a fair die is 3.5, which cannot be rolled. That is fine.
E[X squared] is not E[X] squared. For a fair die E[X squared] = 91/6 = 15.167 but (E[X]) squared = 12.25. Their difference, 2.917 = 35/12, is exactly the variance.
Variance is not a probability and its unit is the square of X's unit. It can be larger than 1.
A short simulation can sit far from E[X]. Seeing a few draws is not evidence the formula is wrong.

Chapter context

Probability quantifies uncertainty relative to a model of possible outcomes. Conditioning changes the relevant population. Before inserting numbers into a formula, say which event is given and which event is being predicted.

TRY IT YOURSELF

Part 1. Do it hands-on

Shape a loaded die with the weight sliders. The checks use exact integer arithmetic for the mean and the exact variance formula.

weight of 11weight of 21weight of 31weight of 41weight of 51weight of 61

probability of a face = its weight / 6

E[X] = sum of face x probability = 3.5

Var(X) = average of (face - 3.5)^2 = 2.917, sd = 1.708

To do: Weight the faces so that the expected roll E[X] is exactly 4.5
To do: Keep E[X] at exactly 3.5 but make the variance at least 6
To do: Keep E[X] at exactly 3.5 but make the variance at most 1

Part 2. Do it in code

Write expectation(values, probs) and variance(values, probs) from the definitions E[X] = sum x p and Var(X) = E[(X - E[X])^2]. Raise ValueError if the lists differ in length or probs does not sum to 1 (tolerance 1e-9).
Write simulate_mean(values, probs, n, seed): draw n values with rng = random.Random(seed) and rng.choices(values, weights=probs, k=n), and return their average. n must be at least 1.
Write running_means(samples): the list whose i-th entry is the average of the first i + 1 samples.

import random

def expectation(values, probs):
    pass

def variance(values, probs):
    pass

def simulate_mean(values, probs, n, seed):
    pass

def running_means(samples):
    pass

die = [1, 2, 3, 4, 5, 6]
fair = [1 / 6] * 6
print(expectation(die, fair))   # 3.5

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Find the mean of a fair die numbered one through six.

A3, because a die has six faces and the middle face is the most typical roll.B3.5, which need not itself be a possible roll.C4, because the average must be one of the faces that can actually be rolled.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B)},\quad E[X]=\sum_x xP(X=x).

\operatorname{Var}(X)=E[(X-E[X])^2],\quad p_i=\frac{e^{z_i-m}}{\sum_j e^{z_j-m}},\;m=\max_jz_j.

Bayes' denominator must be nonzero. The softmax normalization runs over the candidate classes.

Worked example

In a fictional population, illness prevalence is 1%, sensitivity is 90% and the false-positive rate is 10%. True positives account for 0.01×0.9=0.009 of the population; false positives account for 0.99×0.1=0.099. Among positives, the illness fraction is 0.009/0.108=1/12≈8.33%. This is an arithmetic example, not medical guidance.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import exp, log, isclose
posterior = .01*.9 / (.01*.9 + .99*.1)
assert isclose(posterior, 1/12)
scores = [0, log(2), log(3)]
weights = [exp(z-max(scores)) for z in scores]
probs = [w/sum(weights) for w in weights]
assert all(isclose(a,b) for a,b in zip(probs, [1/6,2/6,3/6]))

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

P(A given B) and P(B given A) are different quantities. A high uncalibrated model confidence is not evidence of a correct probability model.