ML9.2 · LESSON 2 OF 6

Leakage

Construct a leakage-resistant pipeline and select an evaluation procedure that reflects deployment.

THE IDEA

Information that will not exist at deployment

Leakage means the training or selection process had access to information that the deployed model will not have. The score then measures something easier than the real task, and the mistake only shows after launch. It arrives in four common forms:

Target leakage. A feature that is produced by, or recorded after, the outcome. A refund paid after a customer cancels predicts churn perfectly because it only exists when churn happened.
Time leakage. Training on rows from after the moment you predict, or using values that were updated later (a total that includes next month).
Duplicate or related rows on both sides of the split: the same customer, the same image taken twice. The test turns into a memory test.
Fitted preprocessing that used test rows or all the labels (the previous lesson).

200 customers, 31% churned. Best accuracy of a single threshold rule on each feature (always guessing "stays" scores 68.5%):

feature	best one-rule accuracy
support_tickets_last_30d known before the prediction date	70.5%
refund_issued recorded after the customer cancels	95.5%

A feature scores suspiciously well on its own. Honest features are modest.

Duplicates turn a test into a memory test

Here the labels are random coin flips, so nothing can be learned. A model that simply remembers its training rows (1-nearest-neighbour) is then tested on rows of which some are exact copies of training rows.

Test rows that are copies of training rows0%

A 1-nearest-neighbour model trained on 300 rows whose labels are pure coin flips scores 50% on 200 test rows, 0 of which are copies of training rows. With no copies it can only guess: about 50%.

With no copies the score is a coin flip. Every copy it can look up is a free point.

Predict first. Half of the 200 test rows are copies of training rows, and the labels carry no signal at all. What accuracy do you expect?

The fix is to split by the unit that must not be shared: customer, patient, document, device. Compare the ids on both sides before you train.

Time runs one way

If the model will predict the future, the test must come from after the training period. A shuffled split mixes months, so some training rows are from later than the test rows they are graded on.

J
train

F
train

M
train

A
train

M
train

J
test

A
test

S
train

O
train

N
train

D
train

12 (training month, test month) pairs have the training month AFTER the test month: the model is allowed to learn from the future of the rows it is graded on.

Same twelve months, two ways of choosing three test months.

In the shuffled split, 12 of the (training month, test month) pairs run backwards. With a cut at a date, none do. Shuffled splits are fine when rows are genuinely exchangeable; they are wrong when the task is forecasting or when rows from the same period share conditions.

The common mistake, failing

a score of 99.8% should not feel good

features = [..., "refund_issued", ...]         # recorded after cancellation
model.fit(X_train, y_train)
accuracy_score(y_test, model.predict(X_test)) # 0.998
# in production the refund column is always 0 at prediction time: accuracy collapses

The habit: for every column ask when is it known, and by whom; for every split ask what is shared across it; for every fitted step ask which rows did it see.

Chapter context

Practical performance depends on the data boundary around training as much as on the estimator. Every learned preprocessing step is part of the model and must respect the same evaluation split.

TRY IT YOURSELF

Part 1. Leakage hunt

Eleven items from one churn project. Give each a verdict. Some are fine: accusing a clean item is also wrong. You are done when all eleven match the true list.

Code snippets

Snippet 1

scaler = StandardScaler().fit(X)      # X is the whole table
X_scaled = scaler.transform(X)
X_tr, X_te, y_tr, y_te = train_test_split(X_scaled, y, random_state=0)

Snippet 2

# predict next week's demand from this week's data
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, shuffle=True)

Snippet 3

train = df[df.date < "2026-06-01"]
test = df[df.date >= "2026-06-01"]
model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(train[cols], train.y)
score = model.score(test[cols], test.y)

Snippet 4

sel = SelectKBest(f_classif, k=10).fit(X, y)   # picks columns using ALL labels
X_10 = sel.transform(X)
scores = cross_val_score(model, X_10, y, cv=5)

Feature table (target: churned, predicted on the 1st of each month in 2026)

account_age_days

Days between sign-up and the prediction date. Known when the prediction is made.

refund_issued

1 if a refund was paid after the customer cancelled. Target: churned.

lifetime_spend

Total spend up to the day the table was exported (2026-10-01). Predictions are made on the 1st of each month in 2026.

tickets_last_30d

Support tickets opened in the 30 days before the prediction date.

city_churn_rate

Share of customers in the same city who churned, computed over all rows, test rows included.

Rows after the split

Rows after the split

train: customer 1042 (2026-03-01), 1043, 1050
test:  customer 1042 (2026-04-01), 1077, 1090
(customer 1042 appears on both sides with nearly identical attributes)

Row counts

train: 1,400 distinct customer ids, test: 350 distinct customer ids, 0 ids in common.

0 of 11 answered, 0 correct so far. Some items are fine: do not accuse everything.

Part 2. Build the detectors

time_split(times, cutoff) returns (train_idx, test_idx), both ascending int arrays: rows with time < cutoff are training rows, rows with time >= cutoff are test rows.
group_overlap(train_ids, test_ids) returns the sorted list of ids that appear on both sides (each once). Duplicates inside one side do not matter.
duplicate_row_indices(X) returns the ascending list of row numbers whose content equals an earlier row of the 2-D array X. A table with no rows returns [].
suspicious_columns(X, y, threshold=0.95) returns the column numbers whose absolute Pearson correlation with y is above the threshold. A constant column, or a constant y, is never flagged and must not crash.

import numpy as np

def time_split(times, cutoff):
    pass

def group_overlap(train_ids, test_ids):
    pass

def duplicate_row_indices(X):
    pass

def suspicious_columns(X, y, threshold=0.95):
    pass

X = np.array([[1, 2], [3, 4], [1, 2]])
print(duplicate_row_indices(X))    # [2]

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Is normalization computed on the whole dataset before splitting harmless?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

Treat the fitted object as preprocessing composed with prediction: model(x) = estimator(transform(x)). Fit both using the training data only. For an imbalanced binary task, report counts as well as rates so denominators remain visible.

Worked example

Training values [0,2] have mean 1. A test value is 100. If the mean is incorrectly fitted on all three values, it becomes 34. The test set has now changed the transformation applied to training examples. A separate 90-negative/10-positive dataset gives 90% accuracy but zero recall for always-negative predictions.

Reference code

The following example uses the Python standard library and does not call a remote service.

train, test = [0,2], [100]
assert sum(train)/len(train) == 1
assert sum(train+test)/len(train+test) == 34
assert 90/100 == .9
assert 0/10 == 0

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

An excellent test score can reflect leakage. More hyperparameter trials can overfit a fixed validation set. Monitoring confidence alone does not measure real outcome quality.