ML1.5 · LESSON 5 OF 6

Loss and metrics

Define a learning problem, establish a baseline and evaluate it on data not used for fitting.

THE IDEA

Loss is for the learner, a metric is for you

The loss is the number the training algorithm minimises. It must be smooth and computable on every example. A metric is what you report to decide whether the model is good enough, and it can be something else entirely: recall, a business cost, latency. Sometimes they are the same (MSE), often not. Minimising squared error does not directly maximise F1 score on a classification task.

For predicting numbers, two standard scores are MSE (mean of squared errors) and MAE (mean of absolute errors). MAE is in the units of the label; squaring in MSE punishes big misses much harder.

house	true	predicted	error	squared
1	200	210	10	100
2	250	260	10	100
3	300	310	10	100
4	350	360	10	100
5	400	410	10	100
mean			MAE 10	MSE 100

miss on house 5 = 10

Four houses are 10 thousand off. Slide the miss on the last house. MAE grows by a little, MSE explodes.

At a miss of 10, MAE is 10 and MSE is 100. At a miss of 50, MAE is 18 (a typical miss of 18) while MSE is 580. Which one is right depends on whether one big miss is much worse than several small ones.

Accuracy can hide a problem

For classification the simplest metric is accuracy: the fraction of correct predictions. It breaks when classes are imbalanced. If 5% of cases are positive (a rare disease, fraud), a model that never flags anyone is 95% accurate and finds nothing. Recall asks: of the true positives, how many did we find? Precision asks: of the cases we flagged, how many were truly positive?

200 synthetic cases, 10 of them positive (5%). The model gives each case a score; a case is flagged when its score reaches the threshold.

	flagged	not flagged
truly positive	5	5
truly negative	0	190

threshold = 0.8

accuracy = 97.5 % recall = 50 % (positives found) precision = 100 % always-no accuracy = 95 %

High accuracy, yet this misses many positives.

200 cases, 10 positive. Move the threshold. Find where accuracy looks great but recall is poor.

At threshold 0.8 the model flags 5 cases, all correct, so accuracy is 97.5%, higher than doing nothing. Yet recall is only 50%: half the sick cases are missed.

Predict first. A fraud detector is 99.5% accurate on data where 0.5% of transactions are fraud. What can you conclude?

Choose the metric before you look

Chapter context

A learning algorithm chooses a model from data rather than requiring a person to specify every decision rule. The resulting model is useful only if its predictions transfer to the intended future inputs under a clearly defined objective.

TRY IT YOURSELF

Part 1. Do it on the visual

Part A is the five-house table, part B the threshold lab. Both share the tasks below.

house	true	predicted	error	squared
1	200	210	10	100
2	250	260	10	100
3	300	310	10	100
4	350	360	10	100
5	400	410	10	100
mean			MAE 10	MSE 100

miss on house 5 = 10

200 synthetic cases, 10 of them positive (5%). The model gives each case a score; a case is flagged when its score reaches the threshold.

	flagged	not flagged
truly positive	5	5
truly negative	0	190

threshold = 0.8

accuracy = 97.5 % recall = 50 % (positives found) precision = 100 % always-no accuracy = 95 %

High accuracy, yet this misses many positives.

To do: Part A: make one big miss that MAE barely feels but MSE punishes: MAE at most 20 while MSE is over 400
To do: Part B: set the threshold so the model flags nobody: accuracy 95% and recall 0
To do: Part B: find a threshold with recall at least 0.9 and accuracy at least 0.85

Part 2. Do it in code (NumPy)

Write mse(y, y_hat) and mae(y, y_hat): mean squared error and mean absolute error. Raise ValueError for empty or mismatched inputs.
Write accuracy(y, y_hat): the fraction of predictions that equal the labels.
Write recall(y, y_hat) for labels 0/1: of the examples whose true label is 1, the fraction predicted 1. If there are no true 1s, return 0.0.
Write always_negative(y): predict 0 for every example and return (accuracy, recall) of that do-nothing model. Compare it with a model that catches positives: which score looks better?

import numpy as np

def mse(y, y_hat):
    pass

def mae(y, y_hat):
    pass

def accuracy(y, y_hat):
    pass

def recall(y, y_hat):
    pass

def always_negative(y):
    # (accuracy, recall) of predicting 0 for everyone
    pass

y = np.array([1, 0, 1, 1, 0])
y_hat = np.array([1, 0, 0, 1, 0])
print(accuracy(y, y_hat), recall(y, y_hat))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does minimizing squared error directly optimize classification F1?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For labeled data $D=\{(x_i,y_i)\}_{i=1}^n$ , empirical risk is

\hat R(\theta)=\frac1n\sum_i\ell(f_\theta(x_i),y_i).

It estimates performance on the sampled population, not automatically on every deployment population. Accuracy is the fraction of correct predictions when classification is the task.

Worked example

A classifier predicts five held-out labels: [1,0,1,1,0] for actual labels [1,0,0,1,0]. Four are correct, so accuracy is 4/5=0.8. This number says nothing about its training accuracy or behavior on a differently collected test set.

Reference code

The following example uses the Python standard library and does not call a remote service.

actual = [1,0,0,1,0]
predicted = [1,0,1,1,0]
correct = sum(a == b for a,b in zip(actual,predicted))
assert correct == 4
assert correct/len(actual) == .8

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Learning a pattern does not establish causation. Validation data used repeatedly for model selection is no longer an untouched test. A random split is not always an independent split.