ML4.5 · LESSON 5 OF 6

Cross-validation

Compare model flexibility, regularization and validation procedures without using the test set for tuning.

THE IDEA

One split is one roll of the dice

A single train/validation split gives you one number, and it depends on which rows landed on which side. k-fold cross-validation makes better use of the data: cut the rows into k parts (folds), then take turns. Each fold is the validation set once while the other k - 1 folds train the model. You get k scores, report their mean, and read their spread as a warning of how much to trust it. Every row is used for validation exactly once and for training k - 1 times. The cost is k model fits instead of one.

k (number of folds) = 5 shuffle the rows before folding

1.781

0.129

0.153

0.025

0.333

Orange cells are the validation rows of that fold, blue cells train. The 20 rows are in order of x, like a file sorted by date or size. The number is that fold's validation MSE for a degree-3 fit.

mean of the 5 fold scores	0.484
spread (standard deviation of the fold scores)	0.656
fold 1: mean of x on its training rows	0.203
mean of x on all 20 rows (scaler fitted before splitting)	0.003
mean of x on this fold's validation rows	-0.795

Without shuffling, a fold holds one end of the x range, so the model must extrapolate and the scores are far above the noise floor of 0.0625. The mean here is 0.484. Tick shuffle and compare.

Choose k and click a fold label. Orange cells are validation rows, blue cells are training rows. The right side shows what the fitted model looked like for the selected fold.

Predict first. You have 20 rows and ask for k = 25 folds. What should the fold generator do?

Try the shuffle box in the figure. The rows are listed in order of x, as they would be in a file sorted by date or size. Without shuffling and with k = 5, each validation fold is one end of the range, and the model has to extrapolate: the mean fold score is 0.484, with a spread of 0.656. After shuffling it is 0.095 with a spread of 0.065. Shuffling is correct for rows that are exchangeable, and wrong for time series and for grouped data (several rows from the same patient or customer), where you must fold by group or in time order so that the model is never validated on its own past or its own relatives.

Preprocessing belongs inside the fold

Any step that learns something from data (a scaler's mean and standard deviation, a feature selector, an imputer's fill value) must be fitted on the training rows of each fold only. If you fit it on all rows first, the validation rows have already influenced the model. That is leakage. Look at the lower rows of the table in the figure above: the scaler's mean computed on all rows differs from the mean of the fold's training rows. The model is being measured on rows whose statistics it was handed in advance.

For a scaler the damage is usually small. For a step that looks at the target, it can be enormous. Here is a table of 60 rows, 200 random columns, and a target that depends on only 3 of them, weakly. We keep the m columns most correlated with y and fit a least squares model with k = 5 folds:

Select the top features:

folds k:keep m features:

Blue bars: skill of each fold (1 is perfect, 0 is no better than predicting the training mean, negative is worse). Orange line: the CV estimate. Green dashed line: skill of the same recipe on 3000 brand new rows.

CV estimate (pooled over folds)	0.413
fold-to-fold spread	0.144
truth on fresh rows	0.083
estimate minus truth	0.331

Too good to be true: choosing the features with all rows already let the validation rows vote on which columns to keep.

Same data, same model, same folds. The only difference is when the columns are chosen. The green dashed line is the honest answer: the recipe applied to 3000 brand new rows.

With 5 features kept, choosing them before splitting reports a skill of 0.41 while the truth on new rows is 0.08. Choosing inside each fold reports -0.2, which is honest but noisy, because with only 60 rows and 200 candidates the right columns are hard to find. Cross-validation cannot repair leakage that happened before the folds were made. It can only measure what you give it.

The leak-free pattern with scikit-learn names (a Pipeline refits every step per fold)

pipe = Pipeline([("scale", StandardScaler()), ("model", Ridge(alpha=1.0))])
scores = cross_val_score(pipe, X, y, cv=5)   # scaler fitted on each training fold only

Common mistakes

Fitting the scaler, imputer, selector or encoder on all rows before splitting.
Shuffling time series or grouped rows, so the model trains on the future or on the same person it is tested on.
Reading the mean score without the spread. A mean of 0.6 with fold scores from 0.1 to 1.1 is not a stable result.

Chapter context

A model should capture reusable structure without chasing accidental patterns in the training sample. Model selection compares this tradeoff using validation data that resembles future use.

TRY IT YOURSELF

Part 1. Tune it on the plot

The data is generated with exactly 3 informative columns among 200. The truth line is measured on 3000 new rows the pipeline never saw. Every number is computed, not scripted.

Select the top features:

folds k:keep m features:

CV estimate (pooled over folds)	-0.228
fold-to-fold spread	0.185
truth on fresh rows	-0.157
estimate minus truth	-0.071

To do: Expose the leak: select features before splitting and find a setup where the CV estimate beats the fresh-data truth by more than 0.2
To do: Make the estimate honest: select inside each fold and land within 0.15 of the fresh-data truth
To do: Build the honest pipeline for the data you know (3 real signal features): selection inside each fold, k of 5 or more, keep 3 features

Part 2. Build it in code

kfold_indices(n, k) returns a list of k NumPy index arrays: the validation rows of each fold, in order, contiguous like np.array_split (the first n % k folds get one extra row). Raise ValueError if k < 2 or k > n.
cv_mse(X, y, lam, k): for each fold, learn the column means and standard deviations on the TRAINING rows only (a standard deviation of 0 counts as 1), standardize both parts with those numbers, fit ridge_fit (given, with intercept) on the training part and score the validation part. Return the total squared error over all folds divided by n.
best_lambda_cv(X, y, lams, k) returns the lambda with the lowest cv_mse. On a tie return the larger lambda.

import numpy as np

def ridge_fit(X, y, lam):
    # given: X has a first column of ones that is not penalized
    D = np.eye(X.shape[1]); D[0, 0] = 0
    return np.linalg.pinv(X.T @ X + lam * D) @ (X.T @ y)

def kfold_indices(n, k):
    pass

def cv_mse(X, y, lam, k):
    # standardize inside every fold, using training rows only
    pass

def best_lambda_cv(X, y, lams, k):
    pass

print([len(f) for f in kfold_indices(10, 3)] if kfold_indices(10, 3) else "not yet")

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Why fit a scaler inside each fold?

AOtherwise the scaler would be fitted too slowly because every fold needs its own copy of the data.BOtherwise the validation fold influences the preprocessing.COtherwise the folds would end up with different numbers of rows.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A possible objective is

J(w)=\operatorname{MSE}(w)+\lambda\sum_jw_j^2.

For L1 replace the squared-weight sum by $\sum_j|w_j|$ . Here lambda is nonnegative and the bias convention must be specified. Different libraries normalize objectives differently.

Worked example

If MSE=0.1, w=[2,−1] and lambda=0.2, the stated L2 objective is 0.1+0.2×5=1.1. The corresponding L1 objective is 0.1+0.2×3=0.7. These values come from different objectives and should not be compared as if they were the same validation metric.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
w, mse, penalty = [2,-1], .1, .2
assert isclose(mse + penalty*sum(x*x for x in w), 1.1)
assert isclose(mse + penalty*sum(abs(x) for x in w), .7)
assert isclose(sum([.7,.8,.9])/3, .8)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Cross-validation does not repair leakage introduced before folding. More regularization can increase bias. A chosen validation result is not an unbiased final test result.