ML9.1 · LESSON 1 OF 6

Pipelines

Construct a leakage-resistant pipeline and select an evaluation procedure that reflects deployment.

THE IDEA

The model is the whole chain

A trained model is not just the classifier. Real data is missing values, has numbers on very different scales and has text categories, so before the estimator there are preprocessing steps: fill gaps, rescale, turn categories into columns. Each of those steps learns something from data: the average used to fill, the mean and spread used to rescale, the list of categories. That learned part is called the fit. Applying what was learned to new rows is the transform. At prediction time the model is estimator(transform(x)), and the transform must be the very same one the estimator was trained on.

That gives one rule for evaluation: every fitted step is part of the model, so it may only be fitted on training rows. If a step is fitted on rows that later serve as the test set, the test set has helped build the model it is meant to judge. A pipeline is simply the tool that keeps the chain together so this rule is hard to break.

fit once on training rows, transform everything else (scikit-learn style)

pipe = Pipeline([
    ("impute", SimpleImputer(strategy="mean")),
    ("scale", StandardScaler()),
    ("model", LogisticRegression()),
])
pipe.fit(X_train, y_train)      # every step learns from X_train only
pipe.score(X_test, y_test)      # the same fitted steps are applied to X_test

Move the split, watch the numbers

Below is a tiny table: 8 training rows (one income is missing), 2 test rows with large incomes (90 and 120) and a city, Pune, that never appears in training. The slider decides where the split happens. Any step to the left of the bar is fitted on all ten rows; any step to its right only on the eight training rows.

Where does the split happen?2 steps saw test rows

Impute (mean)

fitted on ALL rows (test rows included)

Scale (standardize)

fitted on ALL rows (test rows included)

Encode (one-hot city)

fitted on the training rows only

Model

fitted on the training rows

Learned statistic	Your pipeline	Split first
impute value	46	29.143
scaler mean	46	29.143
scaler spread	31.084	5.71
city columns	Delhi, Mumbai	Delhi, Mumbai
mean of scaled training column	-0.475	0

Income of the 8 training rows: 20, 22, (missing), 27, 30, 32, 35, 38. Test rows: 90 and 120, and a city (Pune) never seen in training. A clean scaler gives the training column mean exactly 0. Anything else means test rows shaped the numbers.

The fitted numbers moved because two test rows (90 and 120) were included. In production those rows do not exist yet, so the model is judged on a transform it cannot have.

Prediction time

Income of one request120

scaled value = (120 - 120) / 1 = 0

Cross-validation (5 folds of 2 values)

Validation fold4

validation values 90, 120
scaler mean 43.9, spread 31.716
validation mean as z-score 1.93 (honest value: 13.01)

Drag the split to the right. The impute value, the scaler mean and the encoder columns all start to depend on rows that do not exist yet at deployment time. The cards underneath show the same idea for single requests and for cross-validation.

Predict first. The missing income is filled with the column mean. Fitted on the 8 training rows only, the fill is 29.143. What is it if the imputer was fitted before the split, on all 10 rows?

At prediction time: reuse, never refit

A single request cannot define a mean or a spread. If you refit the scaler on one row, its mean is that row itself, so the centred value is always 0 whatever the income was. The model would receive the same input for a customer earning 20 and one earning 120. The fitted scaler (mean 29.143, spread 5.71) must be saved with the model and reused: an income of 120 becomes 15.91.

Predict first. A request arrives with income 120. You call fit_transform on that single row. What does the scaled value look like?

The common mistake, failing

The classic bug is scaling the whole table first, then cross-validating. Every fold's validation rows were already used to compute the mean and spread. In the cross-validation card above, the last fold holds the two big incomes (90 and 120). Fitted on all ten values the scaler has mean 43.9 and the fold looks like a z-score of 1.93. Fitted honestly on the other eight values (mean 28.625) it is 13.01: the leaky scaler already "knew" the range and hid how surprising these rows are.

wrong, then right

X_scaled = StandardScaler().fit_transform(X)          # sees every validation fold
cross_val_score(model, X_scaled, y, cv=5)             # looks fine, is optimistic

cross_val_score(make_pipeline(StandardScaler(), model), X, y, cv=5)   # refits the scaler inside each training fold

Chapter context

Practical performance depends on the data boundary around training as much as on the estimator. Every learned preprocessing step is part of the model and must respect the same evaluation split.

TRY IT YOURSELF

Part 1. Do it hands-on

The pipeline starts in its worst shape: split last, a scaler refitted for each request, and a scaler fitted once before cross-validation. Fix all three. Tasks tick from the computed state.

Where does the split happen?3 steps saw test rows

Impute (mean)

fitted on ALL rows (test rows included)

Scale (standardize)

fitted on ALL rows (test rows included)

Encode (one-hot city)

fitted on ALL rows (test rows included)

Model

fitted on the training rows

Learned statistic	Your pipeline	Split first
impute value	46	29.143
scaler mean	46	29.143
scaler spread	31.084	5.71
city columns	Delhi, Mumbai, Pune	Delhi, Mumbai
mean of scaled training column	-0.475	0

The fitted numbers moved because two test rows (90 and 120) were included. In production those rows do not exist yet, so the model is judged on a transform it cannot have.

Prediction time

Income of one request120

scaled value = (120 - 120) / 1 = 0

Cross-validation (5 folds of 2 values)

Validation fold4

validation values 90, 120
scaler mean 43.9, spread 31.716
validation mean as z-score 1.93 (honest value: 13.01)

To do: Move the split so that no fitted step (impute, scale, encode) has seen a test row
To do: At prediction time, transform a single request with the already-fitted scaler (do not refit)
To do: Cross-validate with the scaler fitted inside each training fold, not once on all ten values

Part 2. Do it in code

fit_pipeline(X_train) learns from the TRAINING rows only and returns a dict with three arrays: fill (the mean of each column, ignoring NaN; 0.0 if a column is all NaN), mean and scale (the mean and population standard deviation of each column after the NaN values are replaced by fill; a standard deviation of 0 is replaced by 1). An empty table raises ValueError.
apply_pipeline(X, params) replaces NaN with params["fill"], then returns (X - mean) / scale. It must not change X or params, and must not learn anything from X.
fold_standardize(X, folds) takes a list of validation index arrays. For each fold, fit on the rows NOT in the fold and return the standardized validation rows. Return the list of results.

import numpy as np

def fit_pipeline(X_train):
    pass

def apply_pipeline(X, params):
    pass

def fold_standardize(X, folds):
    pass

X = np.array([[0.0], [2.0]])
p = fit_pipeline(X)
print(apply_pipeline(np.array([[100.0]]), p))   # 99.0 : the training mean is 1 and the spread is 1

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Should a production prediction refit the scaler on that one request?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

Treat the fitted object as preprocessing composed with prediction: model(x) = estimator(transform(x)). Fit both using the training data only. For an imbalanced binary task, report counts as well as rates so denominators remain visible.

Worked example

Training values [0,2] have mean 1. A test value is 100. If the mean is incorrectly fitted on all three values, it becomes 34. The test set has now changed the transformation applied to training examples. A separate 90-negative/10-positive dataset gives 90% accuracy but zero recall for always-negative predictions.

Reference code

The following example uses the Python standard library and does not call a remote service.

train, test = [0,2], [100]
assert sum(train)/len(train) == 1
assert sum(train+test)/len(train+test) == 34
assert 90/100 == .9
assert 0/10 == 0

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

An excellent test score can reflect leakage. More hyperparameter trials can overfit a fixed validation set. Monitoring confidence alone does not measure real outcome quality.