ML2.2 · LESSON 2 OF 6

Residuals and squared loss

Fit a linear predictor by hand, implement its loss and gradient, then compare with a library solution.

THE IDEA

How wrong is a line?

For each day, the model makes a prediction and the real sale is known. The gap is the residual. In this chapter it is always residual = prediction − target: positive when the line is above the point, negative when it is below. In the picture a residual is the red vertical segment between a point and the line.

To score a whole line we need a single number. Averaging the residuals fails: a line that is 3 too high on one day and 3 too low on another has an average residual of 0, which makes the worst fit look perfect. So we square each residual first, then average. The result is the mean squared error:

The loss

MSE = (1/n) * sum( (w*x_i + b - y_i)^2 )

Each squared residual is literally a square: the orange squares in the picture have side lengths equal to the residuals. Loss is the average area. Two things follow. Squares are never negative, so over and under cannot cancel. And a miss of 3 costs 9 while a miss of 1 costs 1, so one large miss is punished more than several small ones.

slope w = 0.5

intercept b = 3

show squares

MSE = 21.521 mean residual = 2.158 largest miss = 7.95 (point 2), squared 63.20

Drag the line. Red segments are residuals, orange squares are their squares, and the readout is the live loss. Show best fit reveals the line with the smallest MSE.

Why squares, and what the minimum looks like

Squares also make the loss smooth: it has no corners, so later we can use calculus on it. And with a squared loss the best intercept for any slope is the one that makes the mean residual exactly zero. Try it: fix w, then move b until the mean residual reads 0. The MSE is smallest there. At the true minimum for both parameters, the best line has w = 2.033 and b = 0.918 on this data, with MSE 1.881.

Predict first. Two residuals are -1 and 1. What is their mean squared error?

Predict first. A line misses one point by 4 and nobody else. A second line misses four points by 1 each. Which has the larger sum of squared residuals?

The common mistake: a low mean residual is not a good fit

A flat line at the average height has a mean residual of exactly 0, yet it ignores the trend entirely and its MSE equals the variance of the sales. Always look at the squared loss, or at the residuals one by one, not at their sum. Also keep the sign convention in mind: this chapter uses prediction − target. Some books use target − prediction. The MSE is the same, but the sign of every residual, and of the gradient later, flips.

Chapter context

Linear regression fits a weighted sum of features. The model is simple enough to inspect numerically, making it a useful bridge between geometry, optimization and software implementation.

TRY IT YOURSELF

Part 1. Do it with the visual

The readout shows the MSE, the mean residual and the largest miss live. Use them to hit each target.

slope w = 0

intercept b = 0

show squares

MSE = 28.898 mean residual = -0.817 largest miss = -9.70 (point 12), squared 94.09

To do: Get the MSE below 3
To do: Build a clearly bad model whose residuals still average to about zero: mean residual within 0.05 and MSE above 10
To do: Find the minimum: slope within 0.1 and intercept within 0.25 of the best line

Part 2. Do it in code

residuals(y_hat, y) returns prediction minus target, as a numpy array.
mse(y_hat, y) returns the mean of the squared residuals.
best_intercept(w, x, y) returns the intercept b that minimises the MSE of w * x + b for a FIXED slope w. At that b the mean residual is exactly zero.

import numpy as np

def residuals(y_hat, y):
    # prediction minus target
    pass

def mse(y_hat, y):
    # mean of squared residuals
    pass

def best_intercept(w, x, y):
    # the b that makes the mean residual zero for this slope
    pass

print(mse(np.array([0.0, 2.0]), np.array([1.0, 1.0])))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Residuals are -1 and 1; what is their mean squared error?

AZero, because the residuals -1 and 1 cancel when they are averaged.BOne: squaring makes both residuals 1, so the mean is 1 rather than the 0 you get by averaging the raw residuals.CTwo, because the squared residuals 1 and 1 are added without dividing by the count.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

\hat y_i=x_i^Tw+b,\quad L=\frac1n\sum_i(\hat y_i-y_i)^2.

\nabla_w L=\frac2nX^T(Xw+b\mathbf1-y),\quad \partial_bL=\frac2n\sum_i(\hat y_i-y_i).

When invertible, a no-intercept least-squares solution is $(X^TX)^{-1}X^Ty$ . Use a pseudoinverse or suitable solver otherwise.

Worked example

Use x=[1,2], y=[2,4], w=1 and b fixed at zero. Predictions [1,2] give residuals [-1,-2], so MSE=(1+4)/2=2.5. The w gradient is 2/2×(−1×1−2×2)=−5. With eta=0.1, w becomes 1.5 and MSE falls to (0.25+1)/2=0.625.

Reference code

The following example uses the Python standard library and does not call a remote service.

x, y, w = [1,2], [2,4], 1.0
mse = lambda weight: sum((weight*a-b)**2 for a,b in zip(x,y))/len(x)
grad = 2/len(x)*sum((w*a-b)*a for a,b in zip(x,y))
assert mse(w) == 2.5 and grad == -5
w -= .1*grad
assert w == 1.5 and mse(w) == .625

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A low residual does not show that a weight is causal. The closed form is not always a literal matrix inverse in reliable code. Include or exclude bias consistently in derivations.