Experiments and reproducibility
Construct a leakage-resistant pipeline and select an evaluation procedure that reflects deployment.
THE IDEA
Two runs, two numbers, no idea why
Last week a run scored 78.3%. Today, "the same code" scores 62.9%. Without a record you cannot tell whether the model got worse, the data changed, or you are comparing two different experiments. A result is only a fact about a model together with the exact data, split, preprocessing, settings, library versions and code that produced it.
| field | Run A (last week) | Run B (today) |
|---|---|---|
| command | python train.py --split-seed 3 | python train.py --split-seed 7 |
| split seed | 3 | 7 |
| data file | train.csv | train.csv |
| library | fitlib 1.4 | fitlib 2.0 |
| fit_intercept | (default) | (default) |
| test accuracy | 78.3% | 62.9% |
fitlib is a made-up library for this exercise. In it, the default of fit_intercept changed between versions 1.4 and 2.0.
What a seed does, and what it does not
A seed fixes randomness: the shuffle in the split, the initial weights. It makes this code on this data repeatable. It does not freeze the data file, the library's defaults, or your preprocessing. Starting from run A and changing a single item to run B's value moves accuracy by 1.7 points for the seed, 7.4 for the data contents and 13.3 for the library default. The seed is the smallest of the three here.
Predict first. Two runs use the same seed and the same code but the training file was edited in place between them (12 labels corrected). Is the comparison valid?
A run record
Store, with every result, what is needed to repeat it:
- a fingerprint of the data (a hash of its contents, or an immutable version id), not just a file name;
- the split and its seed;
- the preprocessing steps and their settings;
- the model settings as actually used, defaults included (print them, do not assume);
- library versions, and the code revision.
{"data": "9f3a07c1...", "split_seed": 3, "params": {"C": 1.0, "fit_intercept": true},
"libs": {"fitlib": "1.4"}, "code": "a1b2c3d", "test_accuracy": 0.783}Chapter context
Practical performance depends on the data boundary around training as much as on the estimator. Every learned preprocessing step is part of the model and must respect the same evaluation split.
TRY IT YOURSELF
Part 1. Do it hands-on
Run A scored 78.3%; run B scored 62.9%. Find out why, make B repeat A exactly, and write the record that would have prevented the confusion.
| field | Run A (last week) | Run B (today) |
|---|---|---|
| command | python train.py --split-seed 3 | python train.py --split-seed 7 |
| split seed | 3 | 7 |
| data file | train.csv | train.csv |
| library | fitlib 1.4 | fitlib 2.0 |
| fit_intercept | (default) | (default) |
| test accuracy | 78.3% | 62.9% |
fitlib is a made-up library for this exercise. In it, the default of fit_intercept changed between versions 1.4 and 2.0.
- To do: Reveal what the log hides: hash both data files and print the effective model settings
- To do: After revealing the hidden fields, name the one difference that changes the score most when changed alone
- To do: Configure run B so that it reproduces run A exactly (same score, not just close)
- To do: Write the run record: tick exactly the things needed to repeat a run
Part 2. Do it in code
- fingerprint(X) returns hashlib.sha256(repr(X.shape).encode() + str(X.dtype).encode() + np.ascontiguousarray(X).tobytes()).hexdigest(). Same shape, dtype and values give the same string; anything else changes it.
- split_indices(n, test_frac, seed): perm = np.random.default_rng(seed).permutation(n); n_test = int(round(n * test_frac)); return (train, test) as ascending lists of ints, test taken from the start of perm. A test_frac outside [0, 1] raises ValueError. It must not touch the global numpy random state.
- make_record(params, seed, X) returns {"params": a COPY of params, "seed": seed, "data": fingerprint(X), "numpy": np.__version__}.
- diff_records(a, b) returns the sorted list of field names that differ. The params dict is flattened to names like "params.C"; a key present in only one record counts as different. Identical records return [].
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Can changing a dataset silently invalidate a model comparison?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
Treat the fitted object as preprocessing composed with prediction: model(x) = estimator(transform(x)). Fit both using the training data only. For an imbalanced binary task, report counts as well as rates so denominators remain visible.
Worked example
Training values [0,2] have mean 1. A test value is 100. If the mean is incorrectly fitted on all three values, it becomes 34. The test set has now changed the transformation applied to training examples. A separate 90-negative/10-positive dataset gives 90% accuracy but zero recall for always-negative predictions.
Reference code
The following example uses the Python standard library and does not call a remote service.
train, test = [0,2], [100]
assert sum(train)/len(train) == 1
assert sum(train+test)/len(train+test) == 34
assert 90/100 == .9
assert 0/10 == 0
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
An excellent test score can reflect leakage. More hyperparameter trials can overfit a fixed validation set. Monitoring confidence alone does not measure real outcome quality.