ML1.6 · LESSON 6 OF 6

Baselines and overfitting

Define a learning problem, establish a baseline and evaluate it on data not used for fitting.

THE IDEA

Start with a baseline

Before trusting a fancy model, build the dumbest sensible one. For numbers, predict the training mean for every example. For categories, predict the most common class. This baseline gives the score any real model must beat. On our 40 validation houses the mean baseline has MSE 7809.

Predict first. A classifier reports 92% accuracy. 90% of the examples belong to one class. How impressive is it?

Training error always falls. Validation error does not.

A flexible model can bend through every training point, memorising even the noise. That is overfitting: very low training error with poor error on new data. Raise the degree and track both curves.

Blue: the 12 training houses the curve is fitted to. Orange: 40 validation houses the curve never saw. A green curve that leaves the picture has predictions beyond 0 to 440 thousand.

degree (flexibility) = 1

degree 1: 2 fitted numbers training MSE = 369.1 (root 19.2) validation MSE = 248.7 (root 15.8) baseline val = 7808.7

blue training, orange validation (capped at 150). Only degrees you have visited are drawn.

baseline 88.4

this model 15.8

Beats the baseline on validation by 97 percent (in MSE).

Orange dots are 40 validation houses the fit never saw. Raise the degree: training error keeps falling, validation error falls, then turns upward.

Root-MSE (typical miss in thousands): degree 0 gives 88 on validation, the best degree 3 gives 15, and degree 9 gives 640 while its training error is only 8. Choosing by training error alone would pick degree 9, the worst model here.

Reading the gap

Both errors high: underfitting. The model is too simple (degree 0 here).
Training low, validation high: overfitting. Simplify, add data, or regularise.
Suspiciously perfect on validation: suspect leakage, not genius.

Chapter context

A learning algorithm chooses a model from data rather than requiring a person to specify every decision rule. The resulting model is useful only if its predictions transfer to the intended future inputs under a clearly defined objective.

TRY IT YOURSELF

Part 1. Do it on the visual

Move the degree slider. The error curves draw only the degrees you have visited. Then compute the same curves in NumPy.

Blue: the 12 training houses the curve is fitted to. Orange: 40 validation houses the curve never saw. A green curve that leaves the picture has predictions beyond 0 to 440 thousand.

degree (flexibility) = 0

degree 0 (the mean baseline): 1 fitted numbers training MSE = 7549.8 (root 86.9) validation MSE = 7808.7 (root 88.4) baseline val = 7808.7

blue training, orange validation (capped at 150). Only degrees you have visited are drawn.

baseline 88.4

this model 88.4

This is the baseline itself.

To do: Find a model whose validation error is under 60% of the baseline (the mean predictor)
To do: Select the degree with the lowest validation error
To do: Show overfitting: pick a degree where training error is under a quarter of validation error
To do: Sweep the whole range: visit every degree from 0 to 9 to draw the full curve

Part 2. Do it in code (NumPy)

Write baseline_mse(y_train, y_val): predict the mean of y_train for every validation example and return the MSE on y_val.
Write majority_accuracy(y_train, y_val): predict the most common class in y_train for every validation example and return the accuracy on y_val. On a tie, use the smaller label.
Write poly_errors(x_train, y_train, x_val, y_val, max_degree): for each degree 0..max_degree, fit np.polyfit on the training data and return a list of (train_mse, val_mse) pairs, one per degree.
Write best_degree(errors): the degree (list position) with the lowest validation error; on a tie the lowest degree. Then run your code and see where training error and validation error part ways.

import numpy as np

def baseline_mse(y_train, y_val):
    pass

def majority_accuracy(y_train, y_val):
    pass

def poly_errors(x_train, y_train, x_val, y_val, max_degree):
    # [(train_mse, val_mse) for degree 0, 1, ..., max_degree]
    pass

def best_degree(errors):
    # position of the smallest validation error
    pass

rng = np.random.default_rng(3)
x_tr = np.sort(rng.uniform(0, 1, 12))
x_va = np.sort(rng.uniform(0, 1, 40))
f = lambda x: 2 * np.sin(3 * x)
y_tr = f(x_tr) + rng.normal(0, 0.3, 12)
y_va = f(x_va) + rng.normal(0, 0.3, 40)
errs = poly_errors(x_tr, y_tr, x_va, y_va, 8)
for d, (a, b) in enumerate(errs):
    print(d, round(a, 3), round(b, 3))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Is training accuracy alone enough to choose a model?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For labeled data $D=\{(x_i,y_i)\}_{i=1}^n$ , empirical risk is

\hat R(\theta)=\frac1n\sum_i\ell(f_\theta(x_i),y_i).

It estimates performance on the sampled population, not automatically on every deployment population. Accuracy is the fraction of correct predictions when classification is the task.

Worked example

A classifier predicts five held-out labels: [1,0,1,1,0] for actual labels [1,0,0,1,0]. Four are correct, so accuracy is 4/5=0.8. This number says nothing about its training accuracy or behavior on a differently collected test set.

Reference code

The following example uses the Python standard library and does not call a remote service.

actual = [1,0,0,1,0]
predicted = [1,0,1,1,0]
correct = sum(a == b for a,b in zip(actual,predicted))
assert correct == 4
assert correct/len(actual) == .8

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Learning a pattern does not establish causation. Validation data used repeatedly for model selection is no longer an untouched test. A random split is not always an independent split.