ML4.4 · LESSON 4 OF 6

L1 and feature selection

Compare model flexibility, regularization and validation procedures without using the test set for tuning.

THE IDEA

Absolute values instead of squares

The L1 penalty adds the absolute sizes of the weights instead of their squares: J(w) = MSE(w) + lambda * (|w1| + |w2| + ...). This is the lasso. It looks like a small change and behaves very differently: it can set weights exactly to 0. A weight of exactly 0 means the feature is not used, so the lasso doubles as feature selection. For w = [2, -1] the L1 norm is 2 + 1 = 3 (the squared L2 norm is 5), so the L1 objective with lambda 0.2 and MSE 0.1 is 0.1 + 0.2 · 3 = 0.7.

w1 = 2w2 = -1lambda = 0.2

MSE (fixed for this demo)	0.1
squared L2 norm, w1² + w2²	5
L1 norm, \|w1\| + \|w2\|	3
L2 objective, MSE + lambda · squared L2	1.1
L1 objective, MSE + lambda · L1	0.7

With w = [2, -1] and lambda = 0.2 the L2 objective is 0.1 + 0.2 · 5 = 1.1 and the L1 objective is 0.1 + 0.2 · 3 = 0.7. They are different objectives, not two scores of one model to compare.

The same arithmetic as before; drag the weights and compare the two objectives.

Diamonds have corners

Here is the geometric reason. Instead of adding a penalty, think of limiting the weights to a budget: the L1 budget |w1| + |w2| ≤ t is a diamond, the L2 budget w1² + w2² ≤ t² is a disc. The orange ovals are lines of equal loss around the best unconstrained weights; the answer is where an oval first touches the blue region. A disc is smooth, so the touch point is a typical spot on its rim. A diamond has sharp corners that sit exactly on the axes, and ovals often hit a corner first, which gives a weight of exactly 0.

best w1 = 1.8best w2 = 0.4budget t = 1

L1 (lasso): |w1| + |w2| ≤ t

solution w = (1, 0) exact zero

L2 (ridge): w1² + w2² ≤ t²

solution w = (0.906, 0.423)

The orange ovals are lines of equal loss around the unconstrained best point. The answer is the first point of the blue region that an expanding oval touches. A circle has no corners, so the touch point is almost never on an axis. A diamond has corners on the axes, and a tilted oval often reaches a corner first.

Move the unconstrained best point and the budget. In the diamond picture a green dot means a weight landed on an axis. The default setting is one where it does.

Not every setting gives a zero. Move the best point close to the diagonal, or raise the budget until the best point is allowed, and the L1 answer leaves the corner. It is a tendency that grows with the strength of the penalty, not a guarantee.

The soft-threshold rule

With the data scaled, the L1 problem can be solved one weight at a time. For a single weight the answer is the soft threshold: take the plain least squares value z, move it towards 0 by lambda, and stop at 0 if you would cross it:

soft_threshold(z, lambda) = sign(z) * max(|z| - lambda, 0)

lambda = 1
z =   -3  ->  -2
z =   -2  ->  -1
z =   -1  ->  0
z = -0.5  ->  0
z =  0.5  ->  0
z =    1  ->  0
z =    2  ->  1
z =    3  ->  2

Everything with |z| up to lambda becomes exactly 0, and the rest is shrunk by exactly lambda. Ridge instead divides: z becomes z / (1 + lambda) for an orthogonal, scaled problem, which is never exactly 0. Cycling through the weights and applying this rule repeatedly (coordinate descent) solves the full lasso, as you will code in practice.

Paths: L1 against L2

strength = lambda max × 10^-1.5sample of the data

Lasso (L1): paths hit exact zero

Ridge (L2): paths only approach zero

feature	x1	x2	x3	x4	x5	x6	x7	x8	non-zero	validation error
lasso	0.79	1.06	1.2	-0.78	-0.07	-0.06	-0.1	0.02	8	1.112
ridge	0.88	0.98	1.24	-0.79	-0.17	-0.13	-0.17	0.12	8	1.147

x1 and x2 are near copies (correlation about 0.95) and both feed y; x3 and x4 also matter; x5 to x8 are pure noise (dashed). Lowest lasso validation error on this grid: strength 10^-1.5, error 1.112.

Eight features: x1 and x2 are near copies, x3 and x4 matter too, x5 to x8 are noise (dashed). Left: lasso. Right: ridge. Raise the strength and watch the table: the lasso weights turn into exact zeros, the ridge weights never do.

On this sample, once the strength reaches lambda max (the largest correlation between a feature and y, 1.948 here) every lasso weight is exactly 0; just below it the first feature enters. As the strength falls, features join one by one, and the noise features enter last. Ridge at the same strengths keeps all eight weights non-zero.

Predict first. Two features are near copies of each other and both help predict y. What does the lasso tend to do?

Chapter context

A model should capture reusable structure without chasing accidental patterns in the training sample. Model selection compares this tradeoff using validation data that resembles future use.

TRY IT YOURSELF

Part 1. Tune it on the plot

The slider is the penalty strength relative to lambda max, on a log scale from 10^-3 to 1. The table shows exact zeros in green. Features are standardized and the sample is fixed.

strength = lambda max × 10^-3

Lasso (L1): paths hit exact zero

Ridge (L2): paths only approach zero

feature	x1	x2	x3	x4	x5	x6	x7	x8	non-zero	validation error
lasso	0.85	1.06	1.34	-0.84	-0.22	-0.15	-0.19	0.17	8	1.197
ridge	0.85	1.05	1.34	-0.84	-0.23	-0.15	-0.2	0.17	8	1.201

To do: Feature selection: find a strength where x1 to x4 are non-zero and x5 to x8 are exactly 0
To do: Find the smallest strength on the slider where every lasso coefficient is exactly 0
To do: Find the lasso strength with the lowest validation error, or within 3 percent of it
To do: Find a model with at most 4 non-zero coefficients whose validation error is still below 1.4

Part 2. Build it in code

soft_threshold(z, t) shrinks z towards 0 by t and returns exactly 0 when |z| <= t: sign(z) * max(|z| - t, 0). It works on numbers and on arrays, and raises ValueError when t < 0.
lam_max(X, y) is the smallest lambda that sets every lasso coefficient to 0: max over columns of |X^T y| / n.
lasso_cd(X, y, lam, n_iter=200) minimizes (1/(2n)) * ||y - X w||^2 + lam * ||w||_1 by cycling over the columns: for each j, rho = X_j . (residual + X_j * w_j) / n, then w_j = soft_threshold(rho, lam) / (X_j . X_j / n). X and y are already centered. A column of all zeros must stay at weight 0, not become nan.

import numpy as np

def soft_threshold(z, t):
    pass

def lam_max(X, y):
    pass

def lasso_cd(X, y, lam, n_iter=200):
    n, p = X.shape
    w = np.zeros(p)
    # cycle over the columns n_iter times
    return w

print(soft_threshold(np.array([-3.0, -0.5, 0.2, 2.0]), 1.0))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

What is the L1 norm of [2,-1]?

AFive, because 2 squared plus (-1) squared is 4 + 1.BOne, because the weights 2 and -1 add up to 1 without taking absolute values.CThree, because the absolute values 2 and 1 are added.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A possible objective is

J(w)=\operatorname{MSE}(w)+\lambda\sum_jw_j^2.

For L1 replace the squared-weight sum by $\sum_j|w_j|$ . Here lambda is nonnegative and the bias convention must be specified. Different libraries normalize objectives differently.

Worked example

If MSE=0.1, w=[2,−1] and lambda=0.2, the stated L2 objective is 0.1+0.2×5=1.1. The corresponding L1 objective is 0.1+0.2×3=0.7. These values come from different objectives and should not be compared as if they were the same validation metric.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
w, mse, penalty = [2,-1], .1, .2
assert isclose(mse + penalty*sum(x*x for x in w), 1.1)
assert isclose(mse + penalty*sum(abs(x) for x in w), .7)
assert isclose(sum([.7,.8,.9])/3, .8)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Cross-validation does not repair leakage introduced before folding. More regularization can increase bias. A chosen validation result is not an unbiased final test result.