ML2.6 · LESSON 6 OF 6

Scaling and evaluation

Fit a linear predictor by hand, implement its loss and gradient, then compare with a library solution.

THE IDEA

Why scale the features

Here a house price is predicted from its size alone, between 22 and 58 m². Because the feature is large, a small change in w changes every prediction a lot, while a small change in b barely matters next to it. The loss bowl becomes a long, thin, tilted valley: steep across, almost flat along. Gradient descent must use a step small enough not to bounce across the steep direction (0.0006 here at most), but that same step creeps along the flat direction. The ratio of steepest to flattest curvature is 24699 for the raw feature, 17.2 after min-max scaling to [0, 1], and 1.0 after standardizing (subtract the mean, divide by the standard deviation).

learning rate = 1.00e-5

feature: x as it is (m2) steepest / flattest curvature = 1688.482 / 0.068 (ratio 24699.2) largest stable learning rate = 0.0006 status (loss within 0.01 of the minimum): not reached in 10,000 iterations

The plot zooms to the contour through the start point (w = 0, b = 0), so its shape, not its size, shows the conditioning: a round bowl is easy, a thin tilted sliver is slow. Orange dot: after 60 steps.

Compare the contours and the number of iterations to reach the tolerance. On unscaled data no safe rate gets there in 10,000 steps. Standardized, one step with rate 0.5 does it.

Min-max scaling at rate 0.3 needs 97 iterations to get within 0.01 of the minimum loss, and the standardized feature needs 8 at the same rate. Scaling changes how fast the optimizer works, not what the best line is: the predictions of the best fit are identical. (Closed-form solutions do not need it for speed, though badly scaled columns still hurt numerical accuracy.)

Fit the scaler on training data only

The mean and standard deviation are learned from data, so they are part of the model. They must come from the training set only, and then be reused unchanged on validation and test data. If the test set influences the mean, information about it has leaked into training and the test score becomes too optimistic. Zero-variance columns need care too: a constant column has standard deviation 0, so dividing by it gives nan. Replace that std with 1 and the column becomes all zeros.

Evaluation: R squared, RMSE and the residual plot

RMSE is the square root of the MSE, in the units of y. R squared is 1 − (sum of squared residuals) / (sum of squared deviations of y from its mean): 1 is perfect, 0 is no better than always predicting the mean, and it can go negative on new data.

A single number cannot tell you whether the straight line is the right shape. Plot residual against x. A good linear fit leaves a patternless cloud around zero. A U, a hump or a fan shows structure the model missed. Dataset A below is truly linear but noisy: R² = 0.70. Dataset B is clearly curved, yet its R² = 0.93 is higher. The number alone would pick the wrong dataset. Only the residual plot gives it away.

R^2 = 0.696 RMSE = 6.12 sign changes along x: 8 of 19

The residuals are scattered above and below zero with no shape. The line captured the trend.

Switch datasets. The left plot shows the data and the fitted line, the right plot the residuals. Look for a shape.

Predict first. A fitted line has R² = 0.93 but its residual plot is a clear U shape. What should you conclude?

Chapter context

Linear regression fits a weighted sum of features. The model is simple enough to inspect numerically, making it a useful bridge between geometry, optimization and software implementation.

TRY IT YOURSELF

Part 1. Do it with the visual

The status line counts real gradient descent iterations from (0, 0). In the second instrument, mark each dataset with the verdict the residual plot supports.

learning rate = 1.00e-5

Is a straight line the right model?

R^2 = 0.696 RMSE = 6.12 sign changes along x: 8 of 19

Is a straight line the right model for this data?

To do: Use the unscaled data with a learning rate of 0.1 and watch gradient descent diverge
To do: On unscaled data pick the biggest learning rate on the slider that does not diverge, and see it still has not reached the tolerance after 10,000 iterations
To do: Standardize the feature and pick a learning rate that reaches the tolerance in 5 iterations or fewer
To do: Residual plots: mark each dataset as a linear model that is fine, or one that is wrong. All three must be right

Part 2. Do it in code

fit_scaler(X) returns (mean, std) per column of X (population standard deviation, ddof = 0). If a column has zero spread, use 1 for its std so nothing divides by zero.
transform(X, mean, std) returns (X - mean) / std. The test set must use the TRAINING mean and std.
r2(y, y_hat) = 1 - SS_res / SS_tot, and rmse(y, y_hat) = sqrt(mean squared residual). If y is constant, r2 is 1.0 for a perfect fit and 0.0 otherwise.

import numpy as np

def fit_scaler(X):
    # per-column mean and std (std 0 -> 1)
    pass

def transform(X, mean, std):
    # (X - mean) / std
    pass

def r2(y, y_hat):
    pass

def rmse(y, y_hat):
    pass

print(fit_scaler(np.array([[1.0, 10.0], [3.0, 10.0], [5.0, 10.0]])))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Should the test set influence the fitted mean used for scaling?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

\hat y_i=x_i^Tw+b,\quad L=\frac1n\sum_i(\hat y_i-y_i)^2.

\nabla_w L=\frac2nX^T(Xw+b\mathbf1-y),\quad \partial_bL=\frac2n\sum_i(\hat y_i-y_i).

When invertible, a no-intercept least-squares solution is $(X^TX)^{-1}X^Ty$ . Use a pseudoinverse or suitable solver otherwise.

Worked example

Use x=[1,2], y=[2,4], w=1 and b fixed at zero. Predictions [1,2] give residuals [-1,-2], so MSE=(1+4)/2=2.5. The w gradient is 2/2×(−1×1−2×2)=−5. With eta=0.1, w becomes 1.5 and MSE falls to (0.25+1)/2=0.625.

Reference code

The following example uses the Python standard library and does not call a remote service.

x, y, w = [1,2], [2,4], 1.0
mse = lambda weight: sum((weight*a-b)**2 for a,b in zip(x,y))/len(x)
grad = 2/len(x)*sum((w*a-b)*a for a,b in zip(x,y))
assert mse(w) == 2.5 and grad == -5
w -= .1*grad
assert w == 1.5 and mse(w) == .625

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A low residual does not show that a weight is causal. The closed form is not always a literal matrix inverse in reliable code. Include or exclude bias consistently in derivations.