M1.6 · LESSON 6 OF 6

A loss landscape

Evaluate functions, compare growth rates and read a loss curve without confusing inputs and outputs.

THE IDEA

Loss: a score for a parameter

A model has a tunable number, a parameter, here called θ (theta). The loss L(θ) scores how bad that setting is: lower is better. Our toy loss is L(θ) = (θ − 1.2)². It measures the squared distance to the best value 1.2, so the loss at θ = 0 is 1.44, at θ = 2 it is 0.64, and at θ = 1.2 it is 0. Squaring makes distances on either side count the same way. The parameter and its loss are different numbers: θ = 2 is bigger than θ = 0, and its loss is smaller.

theta = -1.5

L(-1.5) = (-1.5 − 1.2)² = 7.29

slope at theta = -5.4 (purple dashed line)

Slope is negative: the loss falls as theta grows, so downhill is to the right.

step size = 0.1

One step: new theta = theta − step size × slope. No steps taken yet.

Move theta and watch the loss and the slope. The purple dashed line shows which way the curve is heading right here.

Slope tells you which way is downhill

The slope at θ is how fast the loss changes as θ increases. Negative slope: the loss falls as θ grows, so go right. Positive slope: go left. Slope near 0: you are on a flat spot. A manual downhill step is: new θ = θ − (step size) × slope. From θ = -1.5 the slope is -5.4; with step size 0.25 the new θ is -0.15.

Predict first. At θ = 3 the slope is 3.6. Which way should a downhill step move θ?

The step size matters. On this bowl, a step size of 0.5 lands on the bottom in one move. A size of 1 jumps exactly over the bottom and back, forever. Above 1 every step lands further away than the last. Below 0.5 you creep in without overshooting. This is the idea behind gradient descent, which you will meet properly after derivatives. For now you are doing it by hand.

A landscape can have more than one dip

Real loss surfaces are not one clean bowl. Switch to the bumpy landscape below, start on the left, and take small downhill steps. You settle into the first dip you meet, where the slope is 0, even though a deeper one exists to the right. Zero slope tells you that you are on a flat spot, not that you are at the lowest point.

theta = -1.5

L(-1.5) = (-1.5 − 1.2)² = 7.29

slope at theta = -5.4 (purple dashed line)

Slope is negative: the loss falls as theta grows, so downhill is to the right.

step size = 0.1

One step: new theta = theta − step size × slope. No steps taken yet.

Smooth bowl versus bumpy landscape. The green ring marks the lowest point in view.

Chapter context

A function maps an input to an output under a stated rule. Graphs let us see the same rule at many inputs; formulas let us calculate a particular output exactly. Both views should describe the same quantity and units.

TRY IT YOURSELF

Part 1. Do it on the plot

Slider moves theta by hand (and clears the step history). The Step downhill button applies one step with the current step size.

theta = -1.5

L(-1.5) = (-1.5 − 1.2)² = 7.29

slope at theta = -5.4 (purple dashed line)

Slope is negative: the loss falls as theta grows, so downhill is to the right.

step size = 0.1

One step: new theta = theta − step size × slope. No steps taken yet.

To do: On the bowl, move theta by hand until the loss is below 0.01
To do: Starting from theta = -1.5 on the bowl, use the Step downhill button (at most 5 steps) to bring the loss below 0.001
To do: Take a downhill step that makes the loss go UP
To do: Switch to the bumpy landscape and stop on a spot where the slope is almost 0 but the loss is not the lowest (at least 0.1 above it)

Part 2. Do it in code

Write loss(theta) = (theta - 1.2) ** 2.
Write slope(theta, h=1e-6): the steepness estimated from the two points theta - h and theta + h: (loss(theta + h) - loss(theta - h)) / (2 * h).
Write descend(theta0, step_size, n_steps): return the list of theta values, starting with theta0, after each of n_steps downhill steps (theta = theta - step_size * slope(theta)). The list has n_steps + 1 entries.

def loss(theta):
    # penalty for a parameter value; smallest at 1.2
    pass

def slope(theta, h=1e-6):
    # steepness from the two points theta - h and theta + h
    pass

def descend(theta0, step_size, n_steps):
    # [theta0, theta after step 1, ..., theta after step n_steps]
    pass

print(loss(0))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

For L(theta)=(theta-1.2)^2, where is the global minimum?

Atheta=0, because the loss is zero when theta is zero.Btheta=1.44, because that is the loss value at theta=0.Ctheta=1.2, with loss zero.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

f:X\to Y,\quad y=f(x),\quad \log_b(b^x)=x.

For the illustrative loss:

L(\theta)=(\theta-1.2)^2\ge0.

Theta is a tunable scalar, 1.2 is the target, and L is a nonnegative output. An exponent and its logarithm undo each other only on their valid domains.

Worked example

Evaluate L(0)=1.44, L(2)=0.64 and L(1.2)=0. The parameter 2 gives a larger input than 0 but a smaller loss. Squaring makes positive and negative distances contribute the same penalty. This is a synthetic landscape, not a claim that real models have one simple valley.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose, log2
loss = lambda theta: (theta - 1.2) ** 2
assert isclose(loss(0), 1.44)
assert isclose(loss(2), 0.64)
assert loss(1.2) == 0
assert log2(8) == 3
assert sum(i*i for i in range(1, 4)) == 14

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

The output of a loss function is not its input parameter. Larger inputs do not always produce larger outputs. Logarithms require an explicit base when comparing information quantities.