ML4.2 · LESSON 2 OF 6

Bias and variance

Compare model flexibility, regularization and validation procedures without using the test set for tuning.

THE IDEA

Ask what would happen with a different sample

Your training set is one random sample. Had you collected a different 15 points from the same source, you would have trained a slightly different model. Two things can then go wrong, and they are different:

Bias is systematic error: the average of all those different models is still off target. A straight line fitted to a wave is wrong in the same way on every sample.
Variance is sensitivity to the sample: the models disagree with each other. A very flexible curve wiggles to wherever this particular sample's noise pushed it.

Together with the noise in the data itself (which no model can predict), they add up: expected squared error = bias² + variance + noise. In the demo we can measure all three, because we know the true curve and can draw as many training sets as we like. In real projects you only have one sample, so you can only reason about this split. Here you can watch it.

Refit the model many times

model degreepoints per training settraining sets drawn = 12

squared bias	0.0176
variance	0.0144
noise (cannot be removed)	0.0625
bias + variance + noise	0.0945

Squared bias is how far the red average sits from the dashed truth. Variance is how far the blue curves wander from the red one. Both are averages over x between -1 and 1 and over the 12 training sets.

degree	1	2	3	4	5	6	7	9
squared bias	0.26	0.139	0.018	0.005	0.002	0.004	0.005	0.086
variance	0.008	0.01	0.014	0.022	0.035	0.054	0.12	0.384
total	0.331	0.211	0.095	0.09	0.099	0.12	0.188	0.533

Each faint blue curve is a fit to a fresh training set. The red curve is their average, the dashed curve is the truth. Raise the number of training sets and change the degree.

Read the picture in two steps. First look at the red average: how far is it from the dashed truth? That distance, squared and averaged, is the bias². Then look at how far the blue curves stray from the red one: that spread is the variance. At degree 1 the blue curves are tightly bunched but all in the wrong place. At degree 9 their average is close to the truth but individual fits fly off in all directions.

n = 15 points per training set, 40 training sets

degree   squared bias   variance   bias + variance + noise
1        0.2616        0.0091     0.3332
3        0.0168        0.0172     0.0966
4        0.0042        0.0244     0.0911
9        0.0147        0.7677     0.8449

Predict first. Going from degree 1 to degree 9, what happens to the squared bias and to the variance?

Predict first. You keep a degree-5 model but give each training set 60 points instead of 15. What happens to its variance?

Common mistakes

Also, bias and variance are properties of the whole learning procedure on a given kind of data, not of a model in isolation, and the "tradeoff" is not a law. More data cuts variance without raising bias. A better representation can cut both. Do not read "bias" as unfairness or "variance" as just the spread of the data: here they are two precise averages over repeated training sets.

Chapter context

A model should capture reusable structure without chasing accidental patterns in the training sample. Model selection compares this tradeoff using validation data that resembles future use.

TRY IT YOURSELF

Part 1. Tune it on the plot

All 40 training sets are used. Choose the degree and the number of points per training set; the table underneath updates with the exact numbers.

model degreepoints per training set

squared bias	0.0168
variance	0.0172
noise (cannot be removed)	0.0625
bias + variance + noise	0.0966

Squared bias is how far the red average sits from the dashed truth. Variance is how far the blue curves wander from the red one. Both are averages over x between -1 and 1 and over the 40 training sets.

degree	1	2	3	4	5	6	7	9
squared bias	0.262	0.14	0.017	0.004	5.7e-4	0.003	0.005	0.015
variance	0.009	0.014	0.017	0.024	0.032	0.053	0.102	0.768
total	0.333	0.217	0.097	0.091	0.095	0.118	0.169	0.845

To do: Find a model dominated by bias: squared bias more than 3 times the variance
To do: Find a model dominated by variance: variance more than 3 times the squared bias
To do: With 15 points per training set, pick the degree whose bias plus variance is lowest (within 3 percent)
To do: Keep a degree of 5 or more but cut its variance in half by using 60 points per training set

Part 2. Build it in code

bias_variance(preds, truth): preds has shape (B, m): the fitted values at m grid points from B different training sets. truth has shape (m,). Return (bias_sq, variance): the mean over grid points of (average prediction - truth) squared, and the mean over sets and grid points of (prediction - average prediction) squared. Raise ValueError when B is 0.
simulate(sets, degree, grid, truth): sets is a list of (x, y) training sets. Fit a degree-d polynomial to each with the helper given in the starter, predict on grid, and return bias_variance of the stack.
Check yourself: bias_sq + variance must equal the mean squared error of the fits against the truth. If your numbers do not add up, one of them is wrong.

import numpy as np

def fit_predict(x, y, degree, grid):
    # given: least squares polynomial, evaluated on grid
    A = np.stack([x ** k for k in range(degree + 1)], axis=1)
    w = np.linalg.lstsq(A, y, rcond=None)[0]
    return np.stack([grid ** k for k in range(degree + 1)], axis=1) @ w

def bias_variance(preds, truth):
    # return (bias_sq, variance)
    pass

def simulate(sets, degree, grid, truth):
    # fit every training set, then bias_variance of the predictions
    pass

print(bias_variance(np.array([[1.0, 2.0], [3.0, 2.0]]), np.array([2.0, 2.0])))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does one split directly measure statistical estimator variance?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A possible objective is

J(w)=\operatorname{MSE}(w)+\lambda\sum_jw_j^2.

For L1 replace the squared-weight sum by $\sum_j|w_j|$ . Here lambda is nonnegative and the bias convention must be specified. Different libraries normalize objectives differently.

Worked example

If MSE=0.1, w=[2,−1] and lambda=0.2, the stated L2 objective is 0.1+0.2×5=1.1. The corresponding L1 objective is 0.1+0.2×3=0.7. These values come from different objectives and should not be compared as if they were the same validation metric.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
w, mse, penalty = [2,-1], .1, .2
assert isclose(mse + penalty*sum(x*x for x in w), 1.1)
assert isclose(mse + penalty*sum(abs(x) for x in w), .7)
assert isclose(sum([.7,.8,.9])/3, .8)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Cross-validation does not repair leakage introduced before folding. More regularization can increase bias. A chosen validation result is not an unbiased final test result.