ML2.1 · LESSON 1 OF 6

A line as a model

Fit a linear predictor by hand, implement its loss and gradient, then compare with a library solution.

THE IDEA

The simplest model: a straight line

Suppose an ice cream stand records, for twelve days, how much warmer or cooler it was than the seasonal average (x, in degrees) and how many hundreds of cones it sold (y). The points drift upward as x grows. A model is a rule that turns an input into a prediction. The simplest rule that follows a drifting cloud is a straight line:

The model

prediction = w * x + b

Two numbers define it. The slope w (also called the weight) says how much the prediction rises when x grows by 1. The intercept b (the bias) is the prediction when x is 0. With w = 2 and b = 1, a day 3 degrees above average gets the prediction 2 × 3 + 1 = 7. Learning means finding good values for w and b. Those two numbers are the parameters, and everything the model knows lives in them.

slope w = 1

intercept b = 0

ask about x = 3

prediction = w * x + b = 1 * 3 + 0 = 3

average squared miss (MSE) = 9.433

Slide w and b and watch the line move through the 12 purple days. Ask about different x values and read the prediction as an equation.

A good line passes through the middle of the cloud

Which line is best? For now use your eyes: the line should rise where the cloud rises and pass roughly through the middle. The readout at the bottom, the MSE, is the average squared distance between the line and the points. A smaller MSE means a better line. Section two of this chapter explains it properly. For this data the best any straight line can do is an MSE of 1.88. Notice that even the best line misses every point a little. The model is a simplification: it ignores everything except the temperature.

Predict first. The model is w = 3, b = −2. What does it predict for x = 4?

Predict first. You shift the intercept b from 1 to 3 and leave w alone. What happens to the line?

Linear means linear in the parameters

The word "linear" refers to w and b, which only ever get multiplied by numbers and added. You could feed the model a transformed input such as x squared and it would still be a linear model, because the prediction is still a weighted sum of fixed inputs. The line is the shape you see when there is one input.

Forcing b = 0 (a line through the origin) is a legitimate but weaker model: here its best slope is 2.025, and its MSE is 2.724 instead of 1.881. You try this below.

Chapter context

Linear regression fits a weighted sum of features. The model is simple enough to inspect numerically, making it a useful bridge between geometry, optimization and software implementation.

TRY IT YOURSELF

Part 1. Do it with the visual

Each task ticks from the real values of w and b and stays ticked once earned.

slope w = 0

intercept b = 0

ask about x = 3

prediction = w * x + b = 0 * 3 + 0 = 0

average squared miss (MSE) = 28.898

To do: Set the slope to 2 and the intercept to 1, so the model predicts 7 when x = 3
To do: Move the line until its average squared miss (MSE) is within 0.5 of the best any line can do
To do: Force the line through the origin (intercept 0) and find its best slope, within 0.1

Part 2. Do it in code

predict(w, b, x) returns w * x + b. It must work for one number and for a whole numpy array of x values.
line_through(p, q) takes two (x, y) points and returns (w, b) of the line through both. If both points have the same x, raise ValueError: a vertical line is not a function of x.

import numpy as np

def predict(w, b, x):
    # the line's prediction at x (x may be one number or an array)
    pass

def line_through(p, q):
    # p and q are (x, y) points; return (w, b)
    pass

print(predict(2, 1, 3))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

For w=2,b=1,x=3, what is the prediction?

ASeven, because the prediction is w*x + b = 2*3 + 1.BFive, because only the slope and the intercept are added: 2 + 1 + 3 - 1.CSix, because the prediction adds the slope, the intercept and the input: 2 + 1 + 3.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

\hat y_i=x_i^Tw+b,\quad L=\frac1n\sum_i(\hat y_i-y_i)^2.

\nabla_w L=\frac2nX^T(Xw+b\mathbf1-y),\quad \partial_bL=\frac2n\sum_i(\hat y_i-y_i).

When invertible, a no-intercept least-squares solution is $(X^TX)^{-1}X^Ty$ . Use a pseudoinverse or suitable solver otherwise.

Worked example

Use x=[1,2], y=[2,4], w=1 and b fixed at zero. Predictions [1,2] give residuals [-1,-2], so MSE=(1+4)/2=2.5. The w gradient is 2/2×(−1×1−2×2)=−5. With eta=0.1, w becomes 1.5 and MSE falls to (0.25+1)/2=0.625.

Reference code

The following example uses the Python standard library and does not call a remote service.

x, y, w = [1,2], [2,4], 1.0
mse = lambda weight: sum((weight*a-b)**2 for a,b in zip(x,y))/len(x)
grad = 2/len(x)*sum((w*a-b)*a for a,b in zip(x,y))
assert mse(w) == 2.5 and grad == -5
w -= .1*grad
assert w == 1.5 and mse(w) == .625

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A low residual does not show that a weight is causal. The closed form is not always a literal matrix inverse in reliable code. Include or exclude bias consistently in derivations.