M7.3 · LESSON 3 OF 6

Bias and variance

Connect likelihood, estimation and uncertainty to entropy, cross-entropy, KL divergence and perplexity.

THE IDEA

An estimator is a recipe, so judge it over many repetitions

An estimator is a rule that turns a data set into an estimate. One data set gives one estimate, and you cannot tell from one number whether the rule is good. To judge a rule, imagine repeating the whole experiment many times and look at the pile of estimates. Two properties matter. Bias: does the pile centre on the truth? (bias = average estimate − truth). Variance: how widely does the pile spread?

The classic example is estimating a variance. Draw n values; compute the mean; add up squared deviations from that sample mean. Divide by n, or by n − 1?

4000 repeated experiments, each draws 10 values from a normal distribution with true variance 4 and applies the estimator.

values per experiment n10

divide by n divide by n - 1 always say 7

	bias	variance	MSE
measured	-0.45	2.799	3.002
exact	-0.4	2.88	3.04

MSE = bias² + variance. Truth: variance 4 (mean 10).

Each experiment draws values from a normal distribution with true variance 4. Switch between the estimators and watch the orange line (average estimate) against the green one (truth).

Why n - 1 is the unbiased choice

The sample mean is fitted to the same data, so the data sit closer to it than to the true mean. Squared deviations from the sample mean are too small on average, by exactly a factor (n − 1) / n. Dividing by n therefore underestimates: the exact bias is −σ² / n. With σ² = 4 and n = 5 that is -0.8; the simulation measures -0.84. Dividing by n − 1 undoes the factor: the exact bias is 0 and the simulation measures -0.05.

Predict first. You use the divide-by-n estimator and increase n from 5 to 40. What happens to its bias?

Unbiased does not mean best

Prefer a low bias? Not always. The total error of an estimator is its mean squared error, MSE = bias² + variance. At n = 5 the divide-by-n estimator has MSE 5.76, while the unbiased divide-by-(n−1) one has 8: the biased one is wrong in a steady direction but wobbles less, and it wins. At the other extreme, "always say 7" never varies at all, yet it is always wrong by 3.

Chapter context

Statistics estimates properties of a population from observations. Information measures describe how surprising those observations are under a probability model. Keep the true or empirical distribution distinct from the model used to describe it.

TRY IT YOURSELF

Part 1. Tune it by hand

Each setting runs 4000 seeded experiments. Tasks tick from the measured bias and the exact table.

4000 repeated experiments, each draws 10 values from a normal distribution with true variance 4 and applies the estimator.

values per experiment n10

divide by n divide by n - 1 always say 7

	bias	variance	MSE
measured	-0.45	2.799	3.002
exact	-0.4	2.88	3.04

MSE = bias² + variance. Truth: variance 4 (mean 10).

To do: Use at least 3 values per experiment and pick the estimator whose measured bias is within 0.25 of zero
To do: Keep the divide-by-n estimator and make its measured bias smaller than 0.15 in size by raising n
To do: Pick the estimator that never changes from experiment to experiment and is always wrong
To do: With exactly 5 values per experiment, pick the estimator among the two real ones that has the lowest mean squared error

Part 2. Do it in code

Write variance_estimate(xs, ddof): the sum of squared deviations from the sample mean divided by (n - ddof). ddof = 0 divides by n, ddof = 1 by n - 1.
It must raise ValueError when there are not more than ddof values.
Write exact_bias(sigma2, n, ddof) returning the exact bias of that estimator for normal data: 0 for ddof = 1 and -sigma2 / n for ddof = 0.
Write empirical_bias(n, reps, ddof, seed): run the experiment with random.Random(seed).gauss(0, 1) data (true variance 1) and return the average estimate minus 1.

import random

def variance_estimate(xs, ddof):
    # (sum of squared deviations from the mean) / (n - ddof)
    pass

def exact_bias(sigma2, n, ddof):
    # exact bias of the estimator on normal data with true variance sigma2
    pass

def empirical_bias(n, reps, ddof, seed):
    # average of reps estimates, minus the true variance 1.0 (data: rng.gauss(0, 1))
    pass

print(variance_estimate([2, 4, 4, 4, 5, 5, 7, 9], 0))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can a model be consistently wrong across many samples?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

H(p)=-\sum_i p_i\log p_i,\quad H(p,q)=-\sum_i p_i\log q_i.

D_{KL}(p\Vert q)=H(p,q)-H(p),\quad \operatorname{PPL}=e^{H(p,q)}.

These expressions use natural logarithms. If q assigns zero probability where p is positive, cross-entropy and KL are infinite.

For n Bernoulli observations with s successes, the maximum-likelihood probability estimate is s/n. For a sample with mean x-bar, the variance estimate dividing squared deviations by n is a different quantity from the usual unbiased sample-variance estimator dividing by n-1. State which one is being used.

Worked example

Use p=[0.5,0.5] and q=[0.25,0.75]. Source entropy is ln 2≈0.693147 nats. Cross-entropy is −0.5ln(0.25)−0.5ln(0.75)≈0.836988. KL≈0.143841 nats. Perplexity is exp(0.836988)≈2.309401, larger than the source's effective uncertainty of 2.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log, exp, isclose
p, q = [.5,.5], [.25,.75]
h = -sum(a*log(a) for a in p)
ce = -sum(a*log(b) for a,b in zip(p,q))
assert isclose(h, 0.6931471805599453)
assert isclose(ce, 0.8369882167858358)
assert isclose(ce-h, 0.1438410362258905)
assert isclose(exp(ce), 2.309401076758503)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Likelihood is not the posterior probability of a model. Lower perplexity across different tokenizers is not a clean model comparison. Bias is not removed merely by computing more decimal places.