ML1.3 · LESSON 3 OF 6

Models and fitting

Define a learning problem, establish a baseline and evaluate it on data not used for fitting.

THE IDEA

A model is a function with knobs

A model is a function from inputs to a prediction. It has settings called parameters that you do not choose by hand: fitting (training) finds good values from data. The simplest useful model has one feature and two parameters: prediction = w × area + b. Here w is the slope and b is the intercept. The input area is not a parameter; it changes with every house.

How do we know a set of parameters is good? We measure the loss: how far predictions are from the true labels on the training data. The usual choice for numbers is the mean squared error, MSE.

One prediction, one squared error

area = 100, true price = 290, w = 2.0, b = 60
prediction = 2.0 * 100 + 60 = 260
error      = 260 - 290 = -30
squared    = 900          # MSE is the average of this over all houses

slope w = 0intercept b = 0

prediction = 0 * area + 0 training MSE = 68143.7 typical miss (root) = 261 thousand mean baseline MSE = 7549.8 best line MSE = 369.1

Not yet better than the do-nothing average.

Twelve training houses (blue). Move w and b by hand. Red lines are the errors; MSE is the average of their squares. Beat the mean baseline, then get near the best line.

Fitting is a search for the lowest loss

You just did the search by hand. A learning algorithm does it automatically. For a straight line there is even a formula: with the 12 training houses the best line is about price = 2.35 × area − 14.7, with a training MSE of 369, a typical miss of about 19 thousand. The algorithm (least squares) and the fitted parameters (w, b) are separate things: the same algorithm on different data gives different parameters.

The training loop in words 1 of 5

Step 1. Pick starting parameters (for example w = 0, b = 0).

Predict first. In y = w x + b, which of these are fitted parameters?

More knobs, more flexibility

A line can only be straight. Letting the model use area², area³ and so on adds parameters and lets the curve bend. The degree controls how many: degree d has d + 1 parameters. Degree 0 is one number, the average. Watch the training loss as you raise the degree.

Blue: the 12 training houses the curve is fitted to. A green curve that leaves the picture has predictions beyond 0 to 440 thousand.

degree (flexibility) = 1

degree 1: 2 fitted numbers training MSE = 369.1 (root 19.2)

The same 12 houses. Raise the degree: the curve bends toward the dots and the training MSE keeps shrinking.

Training MSE: degree 0 is 7550, degree 1 is 369, degree 3 is 263, degree 9 is 61. It never rises. But a low training loss only says the model fits these houses. Whether that carries over to new ones is the subject of the next lessons.

Chapter context

A learning algorithm chooses a model from data rather than requiring a person to specify every decision rule. The resulting model is useful only if its predictions transfer to the intended future inputs under a clearly defined objective.

TRY IT YOURSELF

Part 1. Do it on the visual

Slide w and b to reduce the training MSE, then write the model, the loss and the closed-form fit in NumPy.

slope w = 0intercept b = 0

prediction = 0 * area + 0 training MSE = 68143.7 typical miss (root) = 261 thousand mean baseline MSE = 7549.8 best line MSE = 369.1

Not yet better than the do-nothing average.

To do: Beat the do-nothing guess: get the training MSE below the mean baseline
To do: Get within 15% of the best possible straight line
To do: Tune it until the training MSE is under 400 (a typical miss of about 20 thousand)

Part 2. Do it in code (NumPy)

Write predict(X, w, b): the model y_hat = X @ w + b for X of shape (n, k) and weights w of shape (k,). Raise ValueError if len(w) is not X.shape[1].
Write mse(y, y_hat): the mean of squared differences. Raise ValueError for empty input or different lengths.
Write fit_line(x, y): the least-squares line y = slope * x + intercept for one feature, using slope = sum((x - mean_x) * (y - mean_y)) / sum((x - mean_x) ** 2). Return (slope, intercept). Raise ValueError if x has fewer than 2 points or all x values are equal.

import numpy as np

def predict(X, w, b):
    # one prediction per row of X
    pass

def mse(y, y_hat):
    # mean squared error
    pass

def fit_line(x, y):
    # (slope, intercept) that minimise the mean squared error
    pass

x = np.array([50.0, 80.0, 110.0, 140.0])
y = np.array([180.0, 252.0, 324.0, 396.0])   # exactly 2.4 * area + 60
print(fit_line(x, y))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

In y=wx+b, which quantities are fitted parameters?

Ax and w; the input and the slope are the values that fitting chooses.Bx and y; the input and the output are what training adjusts.Cw and b; x is an input.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For labeled data $D=\{(x_i,y_i)\}_{i=1}^n$ , empirical risk is

\hat R(\theta)=\frac1n\sum_i\ell(f_\theta(x_i),y_i).

It estimates performance on the sampled population, not automatically on every deployment population. Accuracy is the fraction of correct predictions when classification is the task.

Worked example

A classifier predicts five held-out labels: [1,0,1,1,0] for actual labels [1,0,0,1,0]. Four are correct, so accuracy is 4/5=0.8. This number says nothing about its training accuracy or behavior on a differently collected test set.

Reference code

The following example uses the Python standard library and does not call a remote service.

actual = [1,0,0,1,0]
predicted = [1,0,1,1,0]
correct = sum(a == b for a,b in zip(actual,predicted))
assert correct == 4
assert correct/len(actual) == .8

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Learning a pattern does not establish causation. Validation data used repeatedly for model selection is no longer an untouched test. A random split is not always an independent split.