ML1.4 · LESSON 4 OF 6

Splits and generalization

Define a learning problem, establish a baseline and evaluate it on data not used for fitting.

THE IDEA

Why not test on the data you trained on?

A model that memorises its training examples scores perfectly on them and tells you nothing about new data. So the data is divided into roles:

Training set: used to fit the parameters.
Validation set: used to compare models and settings (like the degree). You look at it repeatedly while making choices.
Test set: touched once, at the end, for an honest estimate of future performance.

Generalization means doing well on data the model has not seen. Because you tune against the validation set, it slowly stops being independent. That is why the test set is kept sealed until the end.

A random split can still leak

Shuffling rows and cutting them into 60/20/20 is the default. It assumes rows are independent. Often they are not: several sales from one account, several photos of one patient, repeated copies of one record. If one account's sales land in both train and test, the model can score well just by finding the sibling it already saw.

30 synthetic sales from 10 accounts, 3 near-identical sales each. train, validation, test. A red outline marks a test sale whose account also sits in train.

A1
val

A1
train

A1
test

A2
val

A2
train

A3
train

A4
train

A5
train

A5
val

A6
train

A6
test

A7
train

A7
val

A7
test

A8
train

A8
test

A9
train

A9
val

A10
train

A10
val

test accounts also in train: 4 (A1, A6, A7, A8) 1-nearest-neighbour test MAE (this shuffle): 3.7 thousand validation MAE (this shuffle): 3.5 thousand average test MAE over 60 shuffles, random split: 16.4 thousand

Test sales have near-copies in train, so the model scores well by memorising. That number will not hold for new accounts.

30 sales from 10 accounts. Compare the two split modes. A 1-nearest-neighbour model (it copies the most similar training sale) is scored on test.

Averaged over 60 shuffles, the random split reports a test error of about 16 thousand. Keeping whole accounts together reports 81 thousand. The second number is the honest one if the model will meet new accounts.

Predict first. Why keep all records from one patient in the same split?

Match the split to deployment

Ask how the model will be used, and make the split mimic that.

Forecasting the future: train on the past, test on later dates. A random split would let the model peek at the future.
New customers: split by customer.
Any scaling or filling of missing values: compute it from the training set only, then apply it to the others. Computing the mean over all rows first lets test data influence training.

Chapter context

A learning algorithm chooses a model from data rather than requiring a person to specify every decision rule. The resulting model is useful only if its predictions transfer to the intended future inputs under a clearly defined objective.

TRY IT YOURSELF

Part 1. Do it on the visual

Use the splitter, then write a seeded three-way split, a leakage detector and a leak-free group split.

30 synthetic sales from 10 accounts, 3 near-identical sales each. train, validation, test. A red outline marks a test sale whose account also sits in train.

A1
val

A1
train

A1
test

A2
val

A2
train

A3
train

A4
train

A5
train

A5
val

A6
train

A6
test

A7
train

A7
val

A7
test

A8
train

A8
test

A9
train

A9
val

A10
train

A10
val

Test sales have near-copies in train, so the model scores well by memorising. That number will not hold for new accounts.

To do: Try both split modes and compare how many test accounts also sit in train
To do: End on a split where no test account also appears in train
To do: Reshuffle at least 3 times: the grouped split must stay leak free every time

Part 2. Do it in code (NumPy)

Write split_indices(n, seed, val_frac, test_frac): shuffle the row numbers 0..n-1 with np.random.default_rng(seed) and return three index arrays (train, val, test). The test set has round(n * test_frac) rows, the validation set round(n * val_frac) rows, train gets the rest. Raise ValueError if a fraction is negative or val_frac + test_frac is 1 or more.
The same seed must give the same split; every row must appear in exactly one set.
Write shared_groups(groups, idx_a, idx_b): the sorted list of group values that occur in the rows idx_a AND in the rows idx_b. That is the leak detector.
Write group_split(groups, seed, test_frac): return (train_idx, test_idx) so that every group sits entirely in train or entirely in test. Use np.unique(groups), shuffle the groups with np.random.default_rng(seed), send round(n_groups * test_frac) groups (at least 1) to test. Both index arrays are sorted ascending. Raise ValueError for fewer than 2 groups.

import numpy as np

def split_indices(n, seed, val_frac, test_frac):
    # (train, val, test) index arrays
    pass

def shared_groups(groups, idx_a, idx_b):
    # sorted list of groups appearing in both index sets
    pass

def group_split(groups, seed, test_frac):
    # (train_idx, test_idx) with whole groups on one side
    pass

# 12 sales from 4 accounts, three sales each
groups = ["a", "a", "a", "b", "b", "b", "c", "c", "c", "d", "d", "d"]
train, val, test = split_indices(12, 0, 0.25, 0.25)
print(shared_groups(groups, train, test))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Why keep records from one patient or account together?

ASplitting related records always guarantees statistical independence.BGrouping records is needed only to sort names alphabetically.CRelated examples across splits can leak information.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For labeled data $D=\{(x_i,y_i)\}_{i=1}^n$ , empirical risk is

\hat R(\theta)=\frac1n\sum_i\ell(f_\theta(x_i),y_i).

It estimates performance on the sampled population, not automatically on every deployment population. Accuracy is the fraction of correct predictions when classification is the task.

Worked example

A classifier predicts five held-out labels: [1,0,1,1,0] for actual labels [1,0,0,1,0]. Four are correct, so accuracy is 4/5=0.8. This number says nothing about its training accuracy or behavior on a differently collected test set.

Reference code

The following example uses the Python standard library and does not call a remote service.

actual = [1,0,0,1,0]
predicted = [1,0,1,1,0]
correct = sum(a == b for a,b in zip(actual,predicted))
assert correct == 4
assert correct/len(actual) == .8

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Learning a pattern does not establish causation. Validation data used repeatedly for model selection is no longer an untouched test. A random split is not always an independent split.