ML4.1 · LESSON 1 OF 6

Underfitting and overfitting

Compare model flexibility, regularization and validation procedures without using the test set for tuning.

THE IDEA

Two ways to be wrong

Imagine 12 noisy measurements of a smooth wave. You fit a polynomial to them. A polynomial has one dial: its degree. Degree 1 is a straight line, degree 3 can bend twice, degree 11 has enough freedom to pass through 12 points exactly. The dial controls flexibility, and both ends are dangerous.

Underfitting. The model is too rigid to follow the real pattern. It is wrong on the points it was trained on and on new points. Degree 1 here has a training error of 0.384 and a validation error of 0.324.
Overfitting. The model is so flexible that it follows the random noise in the sample too. It looks excellent on the training points and falls apart on new ones. Degree 11 here has a training error of 5.7e-6 and a validation error of 322.4.

To tell them apart you need two sets of points. The training points are the ones the model is allowed to look at. The validation points are held back and only used to score the finished fit. The noise in this data has a standard deviation of 0.25, so even a perfect model cannot get below an average squared error of 0.25² = 0.0625. That is the noise floor.

Sweep the dial

polynomial degree = 1training points show validation points

degree 1, 12 training points	training error 0.384	validation error 0.324	gap -0.06

Both errors are high: the model is too stiff to follow the curve. This is underfitting.

Move the degree slider. Left: the fitted curve (blue) against the true curve (dashed grey) with training points (solid) and validation points (hollow). Right: the two errors on a log scale, drawn as you sweep.

The blue training error only goes one way: it never rises as the degree grows, because a more flexible family contains every simpler one. The orange validation error is U-shaped. It falls while the model is learning real structure, bottoms out near degree 4, then climbs as the extra flexibility is spent on noise. You pick the model at the bottom of the U, and you can only see that bottom because the validation points were kept apart.

Predict first. The degree-11 curve has a training error of 5.7e-6, essentially perfect. What does that say about how it will do on new points?

What fixes overfitting

Three levers, all of which you will use in this chapter: reduce flexibility (a lower degree), give the model more data, or penalize wild fits (regularization, lessons 3 and 4). More data really helps. With 60 training points instead of 12, the same degree-11 model has a validation error of 0.077 instead of 322.4:

Degree 11, same dial, different amounts of data

training points   training error   validation error
        12            5.7e-6           322.4
        60            0.052            0.077

The common mistake

Predict first. A model has a training error of 0.40 and a validation error of 0.42. Which is the best next step?

Chapter context

A model should capture reusable structure without chasing accidental patterns in the training sample. Model selection compares this tradeoff using validation data that resembles future use.

TRY IT YOURSELF

Part 1. Tune it on the plot

Four small goals, each ticks when you reach it and stays ticked. The training set can be 12, 20, 40 or 60 points. The model is a polynomial fitted by least squares.

polynomial degree = 6training points show validation points

degree 6, 12 training points	training error 0.056	validation error 0.097	gap 0.041

Both errors are low and close together, near the noise floor. Nothing more to gain from a more flexible model.

To do: Find an underfit model: training error AND validation error both above 0.15
To do: Find an overfit model: training error below 0.03 while validation error is more than 10 times larger
To do: Choose the degree with the lowest validation error, or within 3 percent of it
To do: Give the degree-11 model 60 points instead of 12: validation error must come down below 0.2

Part 2. Build it in code

poly_features(x, degree) returns the design matrix with columns x**0, x**1, ..., x**degree (shape n by degree+1). Raise ValueError for a negative degree or an x that is not 1-D.
fit_poly(x, y, degree) returns the least squares coefficients (use np.linalg.lstsq so it still works when there are fewer points than coefficients). mse(y, y_hat) is the mean squared error.
best_degree(x_tr, y_tr, x_va, y_va, max_degree) fits every degree from 0 to max_degree on the training data and returns the degree with the lowest validation error. On a tie, return the smaller degree.

import numpy as np

def poly_features(x, degree):
    # columns x**0 ... x**degree
    pass

def fit_poly(x, y, degree):
    # least squares coefficients
    pass

def mse(y, y_hat):
    pass

def best_degree(x_tr, y_tr, x_va, y_va, max_degree):
    # lowest validation mse; smaller degree wins a tie
    pass

x = np.linspace(-1, 1, 5)
print(poly_features(x, 2))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Is high error on both training and validation usually evidence of memorization?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A possible objective is

J(w)=\operatorname{MSE}(w)+\lambda\sum_jw_j^2.

For L1 replace the squared-weight sum by $\sum_j|w_j|$ . Here lambda is nonnegative and the bias convention must be specified. Different libraries normalize objectives differently.

Worked example

If MSE=0.1, w=[2,−1] and lambda=0.2, the stated L2 objective is 0.1+0.2×5=1.1. The corresponding L1 objective is 0.1+0.2×3=0.7. These values come from different objectives and should not be compared as if they were the same validation metric.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
w, mse, penalty = [2,-1], .1, .2
assert isclose(mse + penalty*sum(x*x for x in w), 1.1)
assert isclose(mse + penalty*sum(abs(x) for x in w), .7)
assert isclose(sum([.7,.8,.9])/3, .8)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Cross-validation does not repair leakage introduced before folding. More regularization can increase bias. A chosen validation result is not an unbiased final test result.