M5.2 · LESSON 2 OF 6

One descent update

Perform gradient descent updates and explain learning rate, stochastic gradients, momentum and Adam.

THE IDEA

The whole algorithm is one line

Take the loss L(θ) = (θ − 3)². Its derivative is 2(θ − 3). Start at θ = 0, where the derivative is −6: a negative slope, so the loss falls as θ increases. Gradient descent turns that into a rule:

θ_new = θ − η · ∇L(θ)

η (eta) is the learning rate, a number you choose. The minus sign moves against the slope, and the slope also sets how far: steep ground gives a long step, flat ground a short one. With η = 0.25: θ_new = 0 − 0.25 · (−6) = 1.5.

Iterating, one update at a time

Update 1 of 3

gradient at θ = 0: 2·(0 − 3) = -6 θ_new = 0 − 0.25 · (-6) = 1.5 loss 9 → 2.25

Step through the three updates. The numbers are computed, not typed in.

Each step recomputes the gradient at the new position. The gradient shrinks as the ground flattens (−6, −3, −1.5), so the steps shrink too, with no extra rule needed.

Learning rate η0.25

L(x) = (x - 3)² so g = 2·(x - 3) update: x ← x − η·g now x = 0 (press Step)

step	x	gradient	loss
0	0	-6	9

Each update multiplies the distance to the minimum by 1 − 2·1·η = 0.5. Positive and below 1: it creeps in from one side. Limit: η below 1.

Press Step and move the learning rate. The orange dots are your parameter values, drawn on the loss curve.

Predict first. From θ = 0 on L = (θ − 3)² with η = 0.5, where is θ after one update?

The common mistake: the wrong sign

Adding the gradient climbs the hill instead. Starting from the same θ = 0 with the sign flipped (η = −0.25) the positions are 0, -1.5, -3.75, -7.125 and the losses 9, 20.25, 45.563, 102.516: it runs away from the minimum and the loss keeps growing. Another popular bug is to subtract the loss from θ. The loss is a height, the gradient is a slope: only the slope tells you which way is down.

Chapter context

The gradient points toward the steepest local increase under Euclidean geometry. To lower a differentiable loss, a basic optimizer steps in the opposite direction. A local direction says little about how large a safe step can be.

TRY IT YOURSELF

Part 1. Choose the learning rate

Same problem as the lesson: L = (θ − 3)² from θ = 0. Move the slider, step the optimizer and watch.

Learning rate η0.05

L(x) = (x - 3)² so g = 2·(x - 3) update: x ← x − η·g now x = 0 (press Step)

step	x	gradient	loss
0	0	-6	9

Each update multiplies the distance to the minimum by 1 − 2·1·η = 0.9. Positive and below 1: it creeps in from one side. Limit: η below 1.

To do: Choose a learning rate so that a single update lands exactly on the minimum, theta = 3
To do: Choose a learning rate where the first update jumps past the minimum (theta above 3) but the loss after 10 updates is still lower than at the start
To do: Choose a learning rate below 0.5 (no overshoot) that brings the loss under 1 within three updates

Part 2. Write the descent loop

Write gd_step(theta, grad, lr): return a NEW list theta - lr * grad, coordinate by coordinate.
Write gradient_descent(grad_fn, theta0, lr, steps): apply gd_step repeatedly and return the whole path as a list of steps + 1 positions, starting with theta0.
Write steps_to_target(loss_fn, grad_fn, theta0, lr, target, max_steps): the number of updates after which the loss first drops to target or below (0 if it already does), or None if that does not happen within max_steps.

def gd_step(theta, grad, lr):
    # return a new list: theta - lr * grad
    pass

def gradient_descent(grad_fn, theta0, lr, steps):
    # return [theta0, theta1, ..., theta_steps]
    pass

def steps_to_target(loss_fn, grad_fn, theta0, lr, target, max_steps):
    # number of updates until loss <= target, or None
    pass

# worked example from the lesson: L = (theta - 3)**2, eta = 0.25
print(gradient_descent(lambda t: [2 * (t[0] - 3)], [0.0], 0.25, 3))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Starting at theta=0 for L=(theta-3)^2 with eta=0.25, find the next theta.

AThe gradient is -6, so theta becomes 1.5.B3, because one step with eta=0.25 lands on the minimum.C-1.5, because the gradient is added instead of subtracted.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

\theta_{t+1}=\theta_t-\eta\nabla L(\theta_t).

Adam uses $m_t=\beta_1m_{t-1}+(1-\beta_1)g_t$ and $v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2$ , coordinatewise. Divide by $1-\beta_1^t$ and $1-\beta_2^t$ before using $\hat m_t/(\sqrt{\hat v_t}+\epsilon)$ in the update.

For the special quadratic $L(\theta)=(\theta-c)^2$ , the residual follows $\theta_{t+1}-c=(1-2\eta)(\theta_t-c)$ . Starting away from c, convergence requires $|1-2\eta|<1$ , or $0<\eta<1$ . This interval depends on this particular curvature and loss normalization.

Worked example

For L=(theta−3)^2, start at 0 and use eta=0.25. The gradients are −6, −3 and −1.5. Three updates produce theta values 1.5, 2.25 and 2.625. Their losses are 2.25, 0.5625 and 0.140625. A fixed learning rate works here because the example is a simple quadratic.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
theta = 0.0
positions = []
for _ in range(3):
    theta -= 0.25 * 2*(theta-3)
    positions.append(theta)
assert positions == [1.5, 2.25, 2.625]
assert isclose((theta-3)**2, 0.140625)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Subtracting loss instead of its gradient is not gradient descent. Squared gradients in Adam are coordinatewise. A zero gradient may identify a saddle point or maximum, not just a minimum.