Monitoring and drift
Construct a leakage-resistant pipeline and select an evaluation procedure that reflects deployment.
THE IDEA
A model keeps meeting new data
After launch the world moves: prices change, a sensor is replaced, a new customer group arrives. Drift means the data the model sees differs from the data it learned from. There are three things to watch:
- Inputs: are feature distributions still like training? Available immediately, no labels needed.
- Outputs: is the share of positive predictions or the score distribution stable?
- Outcomes: is accuracy holding? This needs labels, which often arrive weeks late.
An input shift is a warning, not a measured error. A feature may move without hurting the model, and a model can degrade while every input histogram looks the same. Treat drift alerts as a reason to look, and keep measuring outcomes when labels arrive.
Two ways to compare a window with training
PSI (population stability index) cuts the training values into bins (here deciles), compares the share of the window in each bin with the training share, and sums (actual - expected) * ln(actual / expected). It is 0 when the shares match and grows with the shift. KS distance is the largest gap between the two cumulative distribution functions: 0 for identical samples, 1 for completely separate ones.
Both are noisy with small samples. With 100 values and 10 bins, even unchanged data gives PSI around 0.1 on an average day, so an alert line of 0.03 would fire on all 60 of 60 days. Common rules of thumb put worry somewhere around 0.1 to 0.25, but they are conventions, not laws; tune against your own data volume.
60 alert days, all false.
30 false alarm days before the drift; first real alert 0 days after it started.
25 false alarm days before the drift; first real alert 0 days after it started.
Blue: the statistic for the last 1 day of data against the training reference (100 values per day). Red line: your alert threshold. Orange dashes: when drift really begins.
Predict first. The PSI of a feature crosses your alert line. Does that prove the model's accuracy has dropped?
Tuning an alert is a trade
A low line or a short window reacts quickly but raises false alarms (the one bad-sensor day, day 20, is a spike that vanishes when averaged with its neighbours). A high line or long window is quiet but slow, or misses a slow creep entirely. With a 3-day window and a line of 0.20 the step drift on day 30 is caught 1 day after it starts and the calm stream stays silent; the slow creep from day 25 takes 9 days to be caught. That is a choice about cost: how bad is a late alert, how bad is a false one?
Chapter context
Practical performance depends on the data boundary around training as much as on the estimator. Every learned preprocessing step is part of the model and must respect the same evaluation split.
TRY IT YOURSELF
Part 1. Do it hands-on
The monitor starts with a 1-day window and a line of 0.03, which cries wolf every day. Tune statistic, window and line until one setting handles all three streams.
60 alert days, all false.
30 false alarm days before the drift; first real alert 0 days after it started.
25 false alarm days before the drift; first real alert 0 days after it started.
Blue: the statistic for the last 1 day of data against the training reference (100 values per day). Red line: your alert threshold. Orange dashes: when drift really begins.
- To do: On the calm stream (one bad-sensor day, no real drift) raise no alert at all
- To do: On the step-drift stream (jump on day 30) alert within 3 days of the jump and never before it
- To do: On the gradual-drift stream (slow creep from day 25) alert within 12 days of the start and never before it
- To do: Use ONE setting that satisfies all three streams at once
Part 2. Do it in code
- psi(expected, actual, bins=10): edges = np.quantile(expected, np.linspace(0, 1, bins + 1)) with the first edge set to -inf and the last to +inf, then np.unique (a constant feature collapses to fewer bins). Count both samples with np.histogram into those edges, divide by their lengths, floor every proportion at 1e-4 (an empty bin must not give log(0)), and return sum((a - e) * ln(a / e)). An empty sample raises ValueError.
- ks_statistic(a, b): the largest absolute gap between the two empirical cumulative distribution functions. An empty sample raises ValueError.
- first_alert(daily_batches, reference, window, threshold, metric=None): day d (0-based, d >= window - 1) pools the last `window` daily batches. Return the first d where metric(reference, pooled) > threshold, or None. The default metric is psi. Days whose pooled data is empty are skipped. window < 1 raises ValueError.
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Is a shifted feature histogram itself a labeled error measurement?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
Treat the fitted object as preprocessing composed with prediction: model(x) = estimator(transform(x)). Fit both using the training data only. For an imbalanced binary task, report counts as well as rates so denominators remain visible.
Worked example
Training values [0,2] have mean 1. A test value is 100. If the mean is incorrectly fitted on all three values, it becomes 34. The test set has now changed the transformation applied to training examples. A separate 90-negative/10-positive dataset gives 90% accuracy but zero recall for always-negative predictions.
Reference code
The following example uses the Python standard library and does not call a remote service.
train, test = [0,2], [100]
assert sum(train)/len(train) == 1
assert sum(train+test)/len(train+test) == 34
assert 90/100 == .9
assert 0/10 == 0
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
An excellent test score can reflect leakage. More hyperparameter trials can overfit a fixed validation set. Monitoring confidence alone does not measure real outcome quality.