Feature engineering
Compare model flexibility, regularization and validation procedures without using the test set for tuning.
THE IDEA
A linear model is linear in its features, not in your raw columns
A linear regression predicts w0 + w1·f1 + w2·f2 + .... The f's are features: whatever columns you give it. They do not have to be the raw inputs. If y depends on x squared, give the model a column of x squared, and the "linear" model draws a parabola. This is feature engineering: reshaping the inputs so the pattern becomes something the model can express. Check each of the three classic cases in the figure. In each, the validation R² (the share of the variation in y that the model explains; 0 is no better than the average) is poor with the raw inputs and good after one added feature.
y = x squared plus noise. Left of zero the line should go down, right of zero up, so a straight line averages to flat.
| features used | x |
|---|---|
| training R² | 0.053 |
| validation R² | -0.169 |
R² is the share of the variation in y that the model explains on the validation points: 0 means no better than the average, 1 means perfect.
- Square. y = x² on x between -2 and 2. A line has R² = -0.17; adding x squared gives 0.96. The curve is symmetric, so the best straight line is flat.
- Interaction. y = 2·x1·x2. Neither x1 nor x2 alone predicts y, because the effect of x1 flips with the sign of x2. R² with both raw features is 0.04; adding the product x1·x2 gives 0.96. In the plot the two colored groups need lines with opposite slopes.
- Log. x is a skewed size from 1 to 100 and each doubling adds the same to y. R² with x is 0.71; with ln(x) it is 0.98. A log turns multiplicative steps into additive ones.
Predict first. A target depends on the product of two features. You fit a plain linear regression on the two raw features. What do you expect?
Features need parameters, and parameters must come from allowed data
Many useful features have numbers inside them: the mean and standard deviation of a standardized column, the 90th percentile used to clip outliers, the average target per category. Those numbers are learned, so the rule from cross-validation applies: compute them on the training rows only, then apply the same numbers to validation and test rows. A validation column standardized with its own mean always looks centered and hides a shift the real model would face.
mean, std = X_train.mean(axis=0), X_train.std(axis=0)
Z_train = (X_train - mean) / std
Z_valid = (X_valid - mean) / std # NOT recomputed on X_validJudge features the way you judge hyperparameters
Is the new feature worth it? Compare candidate feature sets by validation score (or cross-validation), with the whole pipeline refit for each candidate, and prefer the simpler set when the scores are close. Once you have chosen features and hyperparameters, spend your untouched test set once for the final number. If you keep adjusting features while looking at the test score, the test set has become a second validation set and your final number is optimistic.
Predict first. You build a feature using the mean of y over all training and validation rows, then evaluate on the validation rows. What is wrong?
Chapter context
A model should capture reusable structure without chasing accidental patterns in the training sample. Model selection compares this tradeoff using validation data that resembles future use.
TRY IT YOURSELF
Part 1. Tune it on the plot
Three datasets, each remembers its own feature choice. A fix only counts if it avoids the label-peek feature and stays simple: at most two features more than the minimum that works.
y = x squared plus noise. Left of zero the line should go down, right of zero up, so a straight line averages to flat.
| features used | x |
|---|---|
| training R² | 0.053 |
| validation R² | -0.169 |
R² is the share of the variation in y that the model explains on the validation points: 0 means no better than the average, 1 means perfect.
- To do: Distance from the middle: get validation R squared to 0.85 or more with no label-derived feature
- To do: Two things that matter together: reach 0.9 or more without peeking at the label
- To do: Skewed sizes: reach 0.95 or more without peeking at the label
- To do: Add the label-peek feature to any dataset, watch validation R squared jump, then remove it again
Part 2. Build it in code
- add_features(X): X has two columns x1, x2. Return the table with columns x1, x2, x1*x2, x1**2, x2**2. Raise ValueError if X does not have exactly 2 columns.
- log_feature(x): return log(1 + x) for an array of non-negative values; raise ValueError if any value is negative (log of zero itself is avoided by the +1).
- standardize_fit(X_tr) returns (mean, std) per column from the training rows only, with a std of 0 replaced by 1. standardize_apply(X, mean, std) returns (X - mean) / std. A validation table standardized with the training numbers will NOT have mean exactly 0, and that is correct.
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Can a feature built using future labels leak information?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
A possible objective is
For L1 replace the squared-weight sum by . Here lambda is nonnegative and the bias convention must be specified. Different libraries normalize objectives differently.
Worked example
If MSE=0.1, w=[2,−1] and lambda=0.2, the stated L2 objective is 0.1+0.2×5=1.1. The corresponding L1 objective is 0.1+0.2×3=0.7. These values come from different objectives and should not be compared as if they were the same validation metric.
Reference code
The following example uses the Python standard library and does not call a remote service.
from math import isclose
w, mse, penalty = [2,-1], .1, .2
assert isclose(mse + penalty*sum(x*x for x in w), 1.1)
assert isclose(mse + penalty*sum(abs(x) for x in w), .7)
assert isclose(sum([.7,.8,.9])/3, .8)
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
Cross-validation does not repair leakage introduced before folding. More regularization can increase bias. A chosen validation result is not an unbiased final test result.