M7.1 · LESSON 1 OF 6

Sampling and estimation

Connect likelihood, estimation and uncertainty to entropy, cross-entropy, KL divergence and perplexity.

THE IDEA

A population, a sample and a statistic

Suppose 5000 people use an app and you want to know their average weekly minutes. Measuring everyone is expensive, so you ask a sample of n people and compute their average. The true average over everyone, here 49.6 minutes, is a population parameter. The number computed from your sample is a statistic, and you use it as an estimate of the parameter. Real populations are never fully visible, so the parameter stays unknown. In this lesson we cheat: we generated the whole population with a seeded random generator so that we can compare every estimate with the truth.

The key fact: a different sample gives a different statistic. The sample mean is itself a random number. The distribution of that number over many possible samples is the sampling distribution.

The spread of the sample mean shrinks like 1 over the square root of n

The population is skewed: the individual values have a standard deviation of 30.1. Yet the average of n random people wobbles far less than one person does, because high and low values partly cancel. The spread of the sample mean is about sd / sqrt(n). This is called the standard error.

Above: everyone in the population (5000 values, skewed). Below: 1000 sample means, each from a sample of 4.

sample size n4

random sample convenience sample

population mean = 49.6, spread (sd) = 30.1

spread of the sample mean = 14.7

theory: sd / sqrt(n) = 15.04

centre of the means = 49.4

Move n and watch the blue histogram of 1000 sample means narrow. Compare the measured spread with the theory line. Try n = 1 first, then 4, 16, 64.

Predict first. At n = 25 the spread of the sample mean is about 5.9. Roughly what is it at n = 100?

More data does not fix a biased collection process

Everything above assumed each person was equally likely to be picked. Real data is rarely like that. If you only survey people who answer an in-app pop-up, you mostly hear from heavy users. In the demo, switch to convenience sample: it can only reach the top 60 percent of users. The spread still shrinks as n grows, but the cloud of means settles around 64.8 instead of the true 49.6.

The same trap in code

# small spread, wrong centre
means = [mean(convenience_sample(n=400)) for _ in range(1000)]
# sd(means) is tiny  ->  looks confident
# mean(means) is 65   ->  the truth is 50

Chapter context

Statistics estimates properties of a population from observations. Information measures describe how surprising those observations are under a probability model. Keep the true or empirical distribution distinct from the model used to describe it.

TRY IT YOURSELF

Part 1. Tune it by hand

The simulation uses a fixed seed, so the same settings always give the same numbers. Tasks tick from the measured spread and centre of 1000 sample means.

Above: everyone in the population (5000 values, skewed). Below: 1000 sample means, each from a sample of 4.

sample size n4

random sample convenience sample

population mean = 49.6, spread (sd) = 30.1

spread of the sample mean = 14.7

theory: sd / sqrt(n) = 15.04

centre of the means = 49.4

To do: Random sampling, at most 100 people per sample: make the spread of the sample mean at most one fifth of the population spread
To do: Random sampling, at most 150 people per sample: make the spread at most one tenth of the population spread
To do: Switch to the convenience sample and make the spread under one tenth of the population spread, yet see the centre stay at least 0.2 population spreads away from the true mean

Part 2. Do it in code

Write mean(xs), sample_sd(xs) (divide by n - 1) and standard_error(xs), the estimated spread of the sample mean.
Write spread_of_means(population, n, reps, seed): draw reps samples of size n with replacement using random.Random(seed), and return the standard deviation of the reps sample means.
Empty input and fewer than 2 values must raise ValueError where a spread cannot be estimated.

import random
from math import sqrt

def mean(xs):
    # the average. ValueError for an empty list
    pass

def sample_sd(xs):
    # sample standard deviation: divide the squared deviations by n - 1.
    # ValueError if there are fewer than 2 values
    pass

def standard_error(xs):
    # how much the sample mean would typically wobble: sample_sd / sqrt(n)
    pass

def spread_of_means(population, n, reps, seed):
    # repeat: draw n values with replacement, take the mean. Return the std dev of the reps means
    pass

print(mean([2, 4, 6]))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does a large convenience sample guarantee representativeness?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

H(p)=-\sum_i p_i\log p_i,\quad H(p,q)=-\sum_i p_i\log q_i.

D_{KL}(p\Vert q)=H(p,q)-H(p),\quad \operatorname{PPL}=e^{H(p,q)}.

These expressions use natural logarithms. If q assigns zero probability where p is positive, cross-entropy and KL are infinite.

For n Bernoulli observations with s successes, the maximum-likelihood probability estimate is s/n. For a sample with mean x-bar, the variance estimate dividing squared deviations by n is a different quantity from the usual unbiased sample-variance estimator dividing by n-1. State which one is being used.

Worked example

Use p=[0.5,0.5] and q=[0.25,0.75]. Source entropy is ln 2≈0.693147 nats. Cross-entropy is −0.5ln(0.25)−0.5ln(0.75)≈0.836988. KL≈0.143841 nats. Perplexity is exp(0.836988)≈2.309401, larger than the source's effective uncertainty of 2.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log, exp, isclose
p, q = [.5,.5], [.25,.75]
h = -sum(a*log(a) for a in p)
ce = -sum(a*log(b) for a,b in zip(p,q))
assert isclose(h, 0.6931471805599453)
assert isclose(ce, 0.8369882167858358)
assert isclose(ce-h, 0.1438410362258905)
assert isclose(exp(ce), 2.309401076758503)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Likelihood is not the posterior probability of a model. Lower perplexity across different tokenizers is not a clean model comparison. Bias is not removed merely by computing more decimal places.