ML4.3 · LESSON 3 OF 6

L2 regularization

Compare model flexibility, regularization and validation procedures without using the test set for tuning.

THE IDEA

Why big weights are a warning sign

Look at the overfitted curve in the last lesson. To bend sharply between 12 points a degree-9 polynomial needs huge coefficients that nearly cancel each other. Without any penalty, the squared size of the non-intercept coefficients is 4.2e5 for this data; at the best penalty it is 32. Wiggly means large weights. So we change what the model is trying to minimize:

The L2 (ridge) objective

J(w) = MSE(w) + lambda * (w1^2 + w2^2 + ...)

The first term wants a good fit. The second term charges for the squared size of the weights, and lambda (a non-negative number you choose) says how expensive that is. Lambda = 0 is the old model. A huge lambda forces every weight to 0 and leaves a flat line. The usual convention, and the one we use, is not to penalize the intercept: shifting the whole prediction up or down is not a sign of wiggliness.

w1 = 2w2 = -1lambda = 0.2

MSE (fixed for this demo)	0.1
squared L2 norm, w1² + w2²	5
L1 norm, \|w1\| + \|w2\|	3
L2 objective, MSE + lambda · squared L2	1.1
L1 objective, MSE + lambda · L1	0.7

With w = [2, -1] and lambda = 0.2 the L2 objective is 0.1 + 0.2 · 5 = 1.1 and the L1 objective is 0.1 + 0.2 · 3 = 0.7. They are different objectives, not two scores of one model to compare.

Check the arithmetic. With w = [2, -1] the squared L2 norm is 5, and the objective is MSE + lambda times 5.

There is a closed form

For squared error plus an L2 penalty you do not need an iterative search. Setting the slope of J to zero gives w = (X^T X + lambda D)^-1 X^T y. Step through a tiny example:

Step 1 of 4

Data. Three points on a line, x = 0, 1, 2 and y = 1, 3, 5. Each row of X starts with a 1 for the intercept: X = [[1,0],[1,1],[1,2]]. Without a penalty the answer is w = [1, 2], the line y = 1 + 2x.

A useful side effect: even when X^T X is singular (two columns that are copies of each other), X^T X + lambda D is invertible for lambda above 0. Ridge stays well defined where plain least squares has no unique answer. Note also that the answer changes with the data, as it should: with y = 1, 3, 6 instead, the slope goes from 2.5 at lambda 0 to 1.667 at lambda 1.

Sweep lambda

lambda = 0degree 9 polynomial, 12 training points

training error	0.018	validation error	3.042
squared size of the coefficients (not counting the intercept)	4.2e5
best lambda on this grid	1e-3.5 (validation error 0.093)

Slide lambda from 0 to large. Left: the fitted curve. Middle: every coefficient shrinks (signed log scale). Right: the U-shaped validation error with the best lambda circled.

The picture is the mirror image of the degree sweep: strong regularization on the right behaves like a stiff model (underfitting), none on the left behaves like an over-flexible one. The best lambda here is 1e-3.5, with a validation error of 0.093 against 3.04 unregularized. Pick lambda the way you pick degree: by validation error, never by training error (which always prefers lambda = 0).

Predict first. In the path plot, what does a coefficient do as lambda increases?

Scale matters

The penalty adds up squared weights, so it treats a weight of 2 the same whether it multiplies a number in metres or in centimetres. But the weight itself depends on the units: if a length is stored in centimetres instead of metres, the fitted weight is 100 times smaller (0.02 instead of 2), so its penalty is 10 000 times smaller (0.0004 instead of 4) and the same lambda barely touches it.

Predict first. Two features carry the same information, but one is stored with numbers 100 times larger (centimetres instead of metres). With the same lambda, which is shrunk more by ridge?

Chapter context

A model should capture reusable structure without chasing accidental patterns in the training sample. Model selection compares this tradeoff using validation data that resembles future use.

TRY IT YOURSELF

Part 1. Tune it on the plot

A degree-9 polynomial is fitted to 12 points with an L2 penalty. The slider runs from lambda = 0 up to 1e2. Training and validation errors are computed live.

lambda = 1e-1degree 9 polynomial, 12 training points

training error	0.074	validation error	0.126
squared size of the coefficients (not counting the intercept)	5.7
best lambda on this grid	1e-3.5 (validation error 0.093)

To do: Find a lambda that overfits: validation error more than 3 times the best value on the path
To do: Find a lambda that is too strong: training error above 0.3 (the model can no longer fit even the training points)
To do: Pick the lambda with the lowest validation error, or within 3 percent of it
To do: Tame the curve: squared size of the coefficients below 50 while training error stays at or below 0.06

Part 2. Build it in code

ridge_fit(X, y, lam): X already has a first column of ones (the intercept). Return w = (X^T X + lam * D)^-1 X^T y where D is the identity matrix except D[0, 0] = 0, so the intercept is not penalized. Raise ValueError for lam < 0.
It must also work when lam is 0 and the columns of X are exact copies of each other (use np.linalg.pinv instead of np.linalg.inv).
ridge_path(X, y, lams) returns one row of coefficients per lambda. best_lambda(X_tr, y_tr, X_va, y_va, lams) returns the lambda with the lowest validation mean squared error; on a tie pick the larger lambda (the simpler model).

import numpy as np

def ridge_fit(X, y, lam):
    # w = (X^T X + lam D)^-1 X^T y, D = identity but D[0, 0] = 0
    pass

def ridge_path(X, y, lams):
    # array of shape (len(lams), n_columns)
    pass

def best_lambda(X_tr, y_tr, X_va, y_va, lams):
    pass

X = np.array([[1.0, 0.0], [1.0, 1.0], [1.0, 2.0]])
print(ridge_fit(X, np.array([1.0, 3.0, 5.0]), 0.0))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

For w=[2,-1], what is the squared L2 norm?

AFive, because 2 squared plus (-1) squared is 4 + 1.BFour, because only the larger weight is squared and the other is ignored.CThree, because the absolute values 2 and 1 are added.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A possible objective is

J(w)=\operatorname{MSE}(w)+\lambda\sum_jw_j^2.

For L1 replace the squared-weight sum by $\sum_j|w_j|$ . Here lambda is nonnegative and the bias convention must be specified. Different libraries normalize objectives differently.

Worked example

If MSE=0.1, w=[2,−1] and lambda=0.2, the stated L2 objective is 0.1+0.2×5=1.1. The corresponding L1 objective is 0.1+0.2×3=0.7. These values come from different objectives and should not be compared as if they were the same validation metric.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
w, mse, penalty = [2,-1], .1, .2
assert isclose(mse + penalty*sum(x*x for x in w), 1.1)
assert isclose(mse + penalty*sum(abs(x) for x in w), .7)
assert isclose(sum([.7,.8,.9])/3, .8)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Cross-validation does not repair leakage introduced before folding. More regularization can increase bias. A chosen validation result is not an unbiased final test result.