Imbalanced classes
Construct a leakage-resistant pipeline and select an evaluation procedure that reflects deployment.
THE IDEA
Accuracy rewards ignoring the rare class
Fraud, disease, failures: the event you care about is rare. In this data about 5% of rows are positive. A model that answers "negative" to everything is right on 96% of the validation rows and finds none of the positives: recall is 0%. Accuracy is the share of all decisions that were right, so it is dominated by the majority class.
Read a confusion matrix instead. With counts: TP positives found, FN positives missed, FP negatives wrongly flagged, TN negatives correctly passed.
- Recall = TP / (TP + FN): of the real positives, how many did we find?
- Precision = TP / (TP + FP): of the cases we flagged, how many were real?
Always keep the counts next to the rates. "40% recall" of 5 positives and of 5,000 are very different claims.
Three knobs, one trade-off
The model below is a logistic regression on one feature. Trained normally, it flags a positive only when it is more than 50% sure, which for a rare class almost never happens: recall is 12.5% (5 of 40), accuracy 96.2%. Three levers change that.
- Class weights make a missed positive cost more during training.
- Resampling copies positives (or drops negatives) in the training data.
- The decision threshold is the probability above which you flag a case. It can be moved after training without refitting.
| model | accuracy | precision | recall | F1 | TP/FP/FN/TN |
|---|---|---|---|---|---|
| Always say negative | 96% | 0% | 0% | 0 | 0/0/40/958 |
| Your logistic model | 96.2% | 62.5% | 12.5% | 0.21 | 5/3/35/955 |
Validation rows: 998, of which 40 are positive (4%). Training set after your choices: 2002 rows.
Weighting positives by 10 raises recall to 62.5% but precision falls to 16.6%: 126 negatives are flagged to catch 25 positives. Lowering the threshold to 15% gives recall 40% with precision 24.2%. Neither is "correct". The right point depends on what a miss and a false alarm cost.
Predict first. With the same trained model you raise the threshold from 50% to 80%. What happens to recall?
The common mistake, failing
Oversampling the whole table and then splitting copies the same positive into both sides. The validation set then holds copies of training positives. Switch the lab to "Before the split" with oversampling on: validation grows to 1879 rows of which 921 are positive, a class balance that does not exist in the real data, and the metrics describe a task you will never face. Resample inside the training fold only, and keep validation and test rows with the real class ratio.
Also split with the class ratio kept (stratified), otherwise a small random split can leave a fold with almost no positives, or none, and its recall is undefined.
Chapter context
Practical performance depends on the data boundary around training as much as on the estimator. Every learned preprocessing step is part of the model and must respect the same evaluation split.
TRY IT YOURSELF
Part 1. Do it hands-on
The model starts with default settings, no weights and a 50% threshold, so it looks accurate while finding almost nothing. Satisfy all five goals; ticks stay once earned.
| model | accuracy | precision | recall | F1 | TP/FP/FN/TN |
|---|---|---|---|---|---|
| Always say negative | 96% | 0% | 0% | 0 | 0/0/40/958 |
| Your logistic model | 96.2% | 62.5% | 12.5% | 0.21 | 5/3/35/955 |
Validation rows: 998, of which 40 are positive (4%). Training set after your choices: 2002 rows.
- To do: Catch at least half of the real positives on the validation rows (recall of 50% or more)
- To do: Reach recall of 40% or more while keeping precision at 20% or more (do not just flag everyone)
- To do: With the threshold left at 50%, reach 50% recall using class weights alone
- To do: With no weights and no resampling, reach 50% recall by moving the threshold alone
- To do: Oversample the positives, but only inside the training rows, and reach 50% recall
Part 2. Do it in code
- stratified_split(y, test_frac, seed) returns (train_idx, test_idx), both ascending. Handle each class on its own, in sorted class order, with one np.random.default_rng(seed): shuffle the class members, then give the first n_test of them to test, where n_test = int(round(len(members) * test_frac)). If the class has 2 or more members clamp n_test to between 1 and len-1; a class with a single member goes to train.
- class_weights(y) returns {class: n_samples / (n_classes * count_of_class)} for the classes that are present. An empty y gives {}.
- precision_recall(y_true, y_pred) returns (precision, recall) for the positive class 1. When a ratio has a zero denominator (nothing predicted positive, or no true positives exist) return 0.0 for it. Lengths that differ raise ValueError.
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
What recall does an always-negative classifier achieve on a dataset containing positives?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
Treat the fitted object as preprocessing composed with prediction: model(x) = estimator(transform(x)). Fit both using the training data only. For an imbalanced binary task, report counts as well as rates so denominators remain visible.
Worked example
Training values [0,2] have mean 1. A test value is 100. If the mean is incorrectly fitted on all three values, it becomes 34. The test set has now changed the transformation applied to training examples. A separate 90-negative/10-positive dataset gives 90% accuracy but zero recall for always-negative predictions.
Reference code
The following example uses the Python standard library and does not call a remote service.
train, test = [0,2], [100]
assert sum(train)/len(train) == 1
assert sum(train+test)/len(train+test) == 34
assert 90/100 == .9
assert 0/10 == 0
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
An excellent test score can reflect leakage. More hyperparameter trials can overfit a fixed validation set. Monitoring confidence alone does not measure real outcome quality.