Several features
Fit a linear predictor by hand, implement its loss and gradient, then compare with a library solution.
THE IDEA
One weight per feature
Real predictions use many inputs. A house price depends on its size, the number of rooms and its age. The model is still a weighted sum, with one weight per feature plus the intercept:
price = w_size * size + w_rooms * rooms + w_age * age + b
= X @ w + b # X has shape (n_houses, 3), w has shape (3,), the result has shape (n_houses,)Each weight reads as "the change in the prediction when this feature grows by 1 and the others stay fixed". On our 40 synthetic houses the fit gives 3.05 per m², 4.61 per room and -1.52 per year of age (prices in thousands), with intercept 22.9. The recipe used to make the data was 3, 8 and −1.5. The fit recovers it approximately, and the match tightens with more data and less noise.
The shapes carry the whole story. The prediction vector has one entry per house. The weight vector has one entry per column. If w does not match the number of columns, the product is not defined and NumPy raises an error instead of guessing. A mismatch there is a bug in your data pipeline, not something to patch.
Blue bar: fitted coefficient. Orange whisker: smallest and largest value when each one of the 40 houses is left out in turn. Bars are on a common scale up to 3.
40 synthetic houses. Price in thousands. The real recipe: 3 per m2, 8 per room, -1.5 per year of age, plus noise.
When two features say the same thing
Now add a fourth feature: the size in square feet. It carries no new information, because it is 10.7639 times the size in m² (plus a little measurement noise). The model can now explain the price with the m² weight, the sq ft weight, or any mix. The predictions barely change, but the individual weights become arbitrary, and leaving out a single house can swing them wildly. With noise of 0.05 sq ft the m² coefficient moves over a range 2795 times wider than in the clean model, and the coefficients themselves are -189.3 (m²) and 17.87 (sq ft). R² stays 0.980. With no noise at all XᵀX is singular.
Predict first. In the collinear model the R² is as high as in the clean model, but the weights look absurd. What does that tell you?
Chapter context
Linear regression fits a weighted sum of features. The model is simple enough to inspect numerically, making it a useful bridge between geometry, optimization and software implementation.
TRY IT YOURSELF
Part 1. Do it with the visual
R squared and the whiskers are recomputed from real least-squares fits (40 house fits for every whisker) each time you change a tick box.
Blue bar: fitted coefficient. Orange whisker: smallest and largest value when each one of the 40 houses is left out in turn. Bars are on a common scale up to 10.
40 synthetic houses. Price in thousands. The real recipe: 3 per m2, 8 per room, -1.5 per year of age, plus noise.
- To do: Build the model that uses all three real features (size, rooms, age) and no duplicate: R squared at least 0.97
- To do: Add the same size in square feet next to size in m2 and make the size coefficient unstable: its leave-one-out range at least 10 times wider than in the clean model, or no unique solution
- To do: Show that dropping informative features costs accuracy: a model with only size has R squared below 0.95
Part 2. Do it in code
- add_intercept(X) returns X with a column of ones appended as the last column: shape (n, d) becomes (n, d + 1).
- predict_many(X, w, b) returns X @ w + b, one prediction per row. The shapes must agree: if w does not have one entry per column of X, raise ValueError.
- fit_multi(X, y) returns (w, b) from np.linalg.lstsq on the intercept-augmented matrix. It must still give correct predictions when two columns are exact multiples of each other.
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
What is the prediction shape for X of shape (5,3) and w of shape (3,)?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
When invertible, a no-intercept least-squares solution is . Use a pseudoinverse or suitable solver otherwise.
Worked example
Use x=[1,2], y=[2,4], w=1 and b fixed at zero. Predictions [1,2] give residuals [-1,-2], so MSE=(1+4)/2=2.5. The w gradient is 2/2×(−1×1−2×2)=−5. With eta=0.1, w becomes 1.5 and MSE falls to (0.25+1)/2=0.625.
Reference code
The following example uses the Python standard library and does not call a remote service.
x, y, w = [1,2], [2,4], 1.0
mse = lambda weight: sum((weight*a-b)**2 for a,b in zip(x,y))/len(x)
grad = 2/len(x)*sum((w*a-b)*a for a,b in zip(x,y))
assert mse(w) == 2.5 and grad == -5
w -= .1*grad
assert w == 1.5 and mse(w) == .625
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
A low residual does not show that a weight is causal. The closed form is not always a literal matrix inverse in reliable code. Include or exclude bias consistently in derivations.