Search and selection
Construct a leakage-resistant pipeline and select an evaluation procedure that reflects deployment.
THE IDEA
Parameters are learned, hyperparameters are chosen
Training learns parameters (weights). Hyperparameters are settings you fix before training: regularization strength, tree depth, learning rate. You find good values by trying several and scoring each on validation data. That is a search with a budget: each trial costs a full training run.
Grid search tries every combination on a lattice. Random search draws each trial independently. When only some hyperparameters matter much, a grid repeats the important value many times: a 3 x 3 grid tries only 3 distinct values of the important parameter, while 9 random trials try 9.
9 trials, 3 distinct values of the important parameter. Darker is better. Red ring: the winner of this one search. Its noisy validation score is 0.733; its true score is 0.693.
| Over 200 repeated searches | average true score of the winner | average number you would report | optimism |
|---|---|---|---|
| grid, 9 trials | 0.681 | 0.707 | +0.026 |
You are choosing on the test score, so the test set is used up: nothing independent is left to report.
Averaged over 200 repeated searches with 9 trials, grid search ends on a configuration with true score 0.681 and random search on 0.833; random search with 16 trials reaches 0.879. A grid is not always worse (its result depends on whether lattice lines happen to land near the peak: try 16 trials), but it spends the budget less flexibly.
The best of many noisy scores is too good
Each validation score is the true quality plus noise from a finite validation set (here noise of 0.03 per trial). Pick the maximum of many such numbers and you tend to pick a configuration that was lucky as well as good. The more trials, the more luck is available. In the table above, the number you would report from the search (the best validation score) averages 0.843 for random search with 9 trials, while the winner's true score is 0.833: optimism of 0.01.
Predict first. You run 200 trials and quote the best validation accuracy as the expected accuracy in production. Is it an unbiased estimate?
Keep search and estimate apart
Nested validation uses two loops. The inner loop searches hyperparameters using only the outer loop's training part. The outer loop scores the winner on the held-out part. The outer score estimates how well the whole procedure, search included, will do.
inner = GridSearchCV(model, param_grid, cv=3) # search on the outer training part
scores = cross_val_score(inner, X, y, cv=5) # outer folds score the winner
scores.mean() # estimate for the whole procedureChapter context
Practical performance depends on the data boundary around training as much as on the estimator. Every learned preprocessing step is part of the model and must respect the same evaluation split.
TRY IT YOURSELF
Part 1. Do it hands-on
The setup starts with a 9-trial grid, winners picked on the test score, and the best search score reported. Make three decisions. The table shows averages over 200 repeated searches, so one lucky seed cannot pass a task.
9 trials, 3 distinct values of the important parameter. Darker is better. Red ring: the winner of this one search. Its noisy validation score is 0.733; its true score is 0.693.
| Over 200 repeated searches | average true score of the winner | average number you would report | optimism |
|---|---|---|---|
| grid, 9 trials | 0.681 | 0.707 | +0.026 |
You are choosing on the test score, so the test set is used up: nothing independent is left to report.
- To do: Use random search with at most 16 trials and reach an average true score of 0.85 or more over 200 repeated searches
- To do: Choose hyperparameters on validation data, so the final test set is touched once
- To do: Report a score that is not optimistic: the average gap between reported and true score must be under 0.005
Part 2. Do it in code
- sample_configs(bounds, n, seed) returns a list of n dicts. bounds maps a name to (lo, hi). Use rng = np.random.default_rng(seed); for each config draw the names in sorted order with lo + (hi - lo) * rng.random(). n = 0 gives []; a negative n or lo > hi raises ValueError.
- random_search(score_fn, bounds, n, seed) calls score_fn(config) exactly once per sampled config and returns (best_config, best_score, history) where history is the list of (config, score). On a tie the earliest config wins. n < 1 raises ValueError.
- select_by_validation(val_scores, outer_scores) picks the index with the highest validation score (the first one on a tie) and returns (index, its validation score, its outer score). Empty input or lists of different length raise ValueError.
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Is the best score among many validation trials automatically an unbiased future estimate?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
Treat the fitted object as preprocessing composed with prediction: model(x) = estimator(transform(x)). Fit both using the training data only. For an imbalanced binary task, report counts as well as rates so denominators remain visible.
Worked example
Training values [0,2] have mean 1. A test value is 100. If the mean is incorrectly fitted on all three values, it becomes 34. The test set has now changed the transformation applied to training examples. A separate 90-negative/10-positive dataset gives 90% accuracy but zero recall for always-negative predictions.
Reference code
The following example uses the Python standard library and does not call a remote service.
train, test = [0,2], [100]
assert sum(train)/len(train) == 1
assert sum(train+test)/len(train+test) == 34
assert 90/100 == .9
assert 0/10 == 0
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
An excellent test score can reflect leakage. More hyperparameter trials can overfit a fixed validation set. Monitoring confidence alone does not measure real outcome quality.