M6.5 · LESSON 5 OF 6

Common distributions

Calculate conditional probabilities, Bayes updates, expectations, variance and softmax probabilities.

THE IDEA

Pick the distribution that matches how the data is generated

A named distribution is a ready-made story about where observations come from. The Bernoulli distribution is the story of one yes/no trial: success with probability p, failure with 1 - p. Its mean is p and its variance is p(1 - p), largest at p = 0.5. A categorical distribution is the same idea with more than two classes: one probability per class, summing to 1 (softmax, the last lesson, builds one).

mean0sd1

Samples (seed 5):

Normal(mean = 0, sd = 1)

density = exp(-(x - mean)^2 / (2 sd^2)) / (sd sqrt(2 pi))

mean = 0, variance = 1

largest density value = 0.399 (a density is not a probability)

P( ≤ X ≤ ) = 0.683

Pick a distribution. Add samples and see the blue histogram settle onto the orange exact formula. Everything is seeded, so the pictures are reproducible.

Counts, flat ranges and bell curves

Binomial(n, p): the number of successes in n independent Bernoulli(p) trials. P(X = k) = C(n, k) p^k (1 - p)^(n - k). Mean np, variance np(1 - p). For ten fair coin flips, P(exactly 5 heads) = 252/1024 = 0.246, only about a quarter.
Uniform(a, b): every value in the range equally dense, with density 1/(b - a). A uniform on 0 to 0.5 has density 2: densities are not probabilities.
Normal(mean, sd): a bell curve. About 68.3% of the probability lies within 1 sd of the mean, 95.4% within 2 sd and 99.7% within 3 sd.

Predict first. You flip a fair coin 10 times. What is the chance of exactly 5 heads?

For continuous values, ask about intervals

For a continuous variable the probability of any exact value is 0; probability is the area under the curve over an interval. Use the interval boxes in the lab: with the standard normal, P(-1 ≤ X ≤ 1) is 0.683. The samples in the histogram are scaled to density (frequency divided by bin width) so that they sit directly on the curve.

Common mistakes

Using a normal for something that is really a count of successes, or a binary outcome. A normal can give negative or fractional counts. Match the distribution to how observations arise.
Reading a density height as a probability. Only areas are probabilities.
Forgetting the trials must be independent with the same p for a binomial to apply.

Chapter context

Probability quantifies uncertainty relative to a model of possible outcomes. Conditioning changes the relevant population. Before inserting numbers into a formula, say which event is given and which event is being predicted.

TRY IT YOURSELF

Part 1. Do it hands-on

Choose the distribution and its parameters in the lab (the interval boxes start at 40 and 60). Tasks are checked from the exact formulas, not from the sample picture.

mean0sd1

Samples (seed 5):

Normal(mean = 0, sd = 1)

density = exp(-(x - mean)^2 / (2 sd^2)) / (sd sqrt(2 pi))

mean = 0, variance = 1

largest density value = 0.399 (a density is not a probability)

P( ≤ X ≤ ) = 0

To do: Model “the number of heads in 10 flips of a fair coin”
To do: Pick a normal distribution where between 94% and 97% of the probability lies between 40 and 60
To do: Choose a Bernoulli distribution whose variance is exactly 0.21
To do: Choose a uniform distribution whose density is exactly 2 (and note that a density above 1 is fine)

Part 2. Do it in code

Write bernoulli_pmf(k, p) (k is 0 or 1; any other k has probability 0) and binomial_pmf(n, k, p) with math.comb. Raise ValueError if p is outside [0, 1]. Remember p = 0 and p = 1 are legal.
Write normal_pdf(x, mu, sigma) for the normal density. Raise ValueError if sigma is not positive.
Write binomial_samples(n, p, trials, seed): a list of trials counts, each the number of successes in n Bernoulli(p) trials, using random.Random(seed).

import math
import random

def bernoulli_pmf(k, p):
    pass

def binomial_pmf(n, k, p):
    pass

def normal_pdf(x, mu, sigma):
    pass

def binomial_samples(n, p, trials, seed):
    pass

print(binomial_pmf(10, 5, 0.5))   # 252 / 1024 = 0.24609375

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Which distribution models one yes/no outcome?

ABinomial, because it models a single trial with two possible results.BBernoulli.CNormal, because every yes/no outcome can be treated as a bell curve.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B)},\quad E[X]=\sum_x xP(X=x).

\operatorname{Var}(X)=E[(X-E[X])^2],\quad p_i=\frac{e^{z_i-m}}{\sum_j e^{z_j-m}},\;m=\max_jz_j.

Bayes' denominator must be nonzero. The softmax normalization runs over the candidate classes.

Worked example

In a fictional population, illness prevalence is 1%, sensitivity is 90% and the false-positive rate is 10%. True positives account for 0.01×0.9=0.009 of the population; false positives account for 0.99×0.1=0.099. Among positives, the illness fraction is 0.009/0.108=1/12≈8.33%. This is an arithmetic example, not medical guidance.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import exp, log, isclose
posterior = .01*.9 / (.01*.9 + .99*.1)
assert isclose(posterior, 1/12)
scores = [0, log(2), log(3)]
weights = [exp(z-max(scores)) for z in scores]
probs = [w/sum(weights) for w in weights]
assert all(isclose(a,b) for a,b in zip(probs, [1/6,2/6,3/6]))

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

P(A given B) and P(B given A) are different quantities. A high uncalibrated model confidence is not evidence of a correct probability model.