M5.1 · LESSON 1 OF 6

Gradients as directions

Perform gradient descent updates and explain learning rate, stochastic gradients, momentum and Adam.

THE IDEA

Many inputs, one slope per input

A loss L depends on parameters. With two parameters, L(x, y) is a landscape: the height above the point (x, y). The partial derivative ∂L/∂x is the slope you meet if you walk along x and keep y fixed. ∂L/∂y is the same for y. The gradient simply stacks them, always in the same order as the parameters: ∇L = [∂L/∂x, ∂L/∂y]. For L = x² + y² the partials are 2x and 2y, so at (1, 2) the gradient is [2, 4].

Contour plots draw a landscape from above: every line connects points of equal loss, like the height lines on a hiking map. Close lines mean steep ground.

Why the gradient is the steepest direction

Pick any direction as a unit vector u. The slope you feel walking that way is the dot product g · u. A dot product is largest when u points the same way as g, and then it equals the length |g|. So the gradient points in the direction of fastest increase, its length is that fastest slope, and −g / |g| is the direction of fastest decrease. Walking along a contour line, perpendicular to g, changes the loss by zero (to first order).

L = x² + y² point = [3, -2] L = 13 gradient = [6, -4] |g| = 7.211 steepest descent = [-0.832, 0.555]

Direction angle (degrees)0°

u = [1, 0] slope along u = g · u = 6·1 + -4·0 = 6 largest possible slope = |g| = 7.211

Click the plot (or focus it and use the arrow keys) to move the point. Red: the gradient, the direction of steepest increase. Green: its opposite, the direction of steepest decrease. Purple dashes: the direction you choose with the slider.

Round contours. Both coordinates have the same curvature, so the steepest direction points straight at the minimum.

Click anywhere on the plot (or use the arrow keys after focusing it). Red is the gradient, green is the steepest descent direction. Turn the purple direction slider and watch g · u: it peaks when purple lies on red, and it is zero along the contour.

Predict first. For L = x² + y², what is the gradient at (3, −2)?

Steepest does not mean toward the minimum

Switch the plot to Elongated valley and click at [4, 0.5]. The gradient is [0.16, 1]: nearly all of it is the y part, because the valley wall is steep across and almost flat along. The steepest descent direction therefore points almost straight at the valley wall opposite, about 74 degrees away from the direction to the minimum. The gradient is local information: it says which way is downhill right here, not where the bottom is.

The same lesson holds at flat spots. At a point where the gradient is exactly zero you cannot tell by the gradient alone whether you are at a minimum. On the saddle surface (L = x² - y² + 0.25 y⁴) the origin has gradient [0, 0], yet the loss is 0 there while it is −1 at (0, ±√2).

Common mistakes

Mixing up the order of partials. An update subtracts the first entry from the first parameter. If you hand back [∂L/∂y, ∂L/∂x], each parameter moves by the other one's slope:

Wrong order, right numbers

L = x**2 + 3*y**2 + x*y          # at (1, 2)
correct gradient = [2*x + y, 6*y + x] = [4, 13]
swapped gradient = [13, 4]
# x would be pushed by y's slope: the step no longer follows the landscape

Chapter context

The gradient points toward the steepest local increase under Euclidean geometry. To lower a differentiable loss, a basic optimizer steps in the opposite direction. A local direction says little about how large a safe step can be.

TRY IT YOURSELF

Part 1. Read the landscape

Click points and turn the slider. Tasks tick from the real numbers and stay ticked once earned.

L = x² + y² point = [3, -2] L = 13 gradient = [6, -4] |g| = 7.211 steepest descent = [-0.832, 0.555]

Direction angle (degrees)0°

u = [1, 0] slope along u = g · u = 6·1 + -4·0 = 6 largest possible slope = |g| = 7.211

Round contours. Both coordinates have the same curvature, so the steepest direction points straight at the minimum.

To do: On the bowl, click the point where the gradient is exactly [2, 4]
To do: At a point where the gradient is not tiny (length at least 1), turn the direction slider until the slope in that direction is 0. Note how that direction relates to the contour line
To do: On the elongated valley, find a point where the steepest-descent direction points more than 60 degrees away from the direction to the minimum at the origin
To do: On the saddle, click a point where the gradient is exactly zero but the loss is not at its lowest value

Part 2. Compute gradients in code

Write grad_L(x, y) for L(x, y) = x² + 3y² + x·y by taking the two partial derivatives by hand. Return [dL/dx, dL/dy] in that order.
Write numeric_grad(f, p, h=1e-5): estimate every partial derivative of any function f (it takes a list) with the central difference (f(p + h·e_i) − f(p − h·e_i)) / (2h). Do not modify p.
Write descent_direction(g): the unit vector pointing opposite to g (steepest descent). If g is the zero vector, return a vector of zeros.

def grad_L(x, y):
    # gradient of L(x, y) = x**2 + 3*y**2 + x*y, as [dL/dx, dL/dy]
    pass

def numeric_grad(f, p, h=1e-5):
    # central difference for each coordinate; f takes a list
    pass

def descent_direction(g):
    # unit vector opposite to g; all zeros if g is zero
    pass

print(grad_L(1, 2))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

For L(x,y)=x squared plus y squared at (1,2), find the gradient.

A[4,2], because the partial derivatives are listed in the wrong order.B[2,4].C[1,2], because the gradient is just the point itself.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

\theta_{t+1}=\theta_t-\eta\nabla L(\theta_t).

Adam uses $m_t=\beta_1m_{t-1}+(1-\beta_1)g_t$ and $v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2$ , coordinatewise. Divide by $1-\beta_1^t$ and $1-\beta_2^t$ before using $\hat m_t/(\sqrt{\hat v_t}+\epsilon)$ in the update.

For the special quadratic $L(\theta)=(\theta-c)^2$ , the residual follows $\theta_{t+1}-c=(1-2\eta)(\theta_t-c)$ . Starting away from c, convergence requires $|1-2\eta|<1$ , or $0<\eta<1$ . This interval depends on this particular curvature and loss normalization.

Worked example

For L=(theta−3)^2, start at 0 and use eta=0.25. The gradients are −6, −3 and −1.5. Three updates produce theta values 1.5, 2.25 and 2.625. Their losses are 2.25, 0.5625 and 0.140625. A fixed learning rate works here because the example is a simple quadratic.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
theta = 0.0
positions = []
for _ in range(3):
    theta -= 0.25 * 2*(theta-3)
    positions.append(theta)
assert positions == [1.5, 2.25, 2.625]
assert isclose((theta-3)**2, 0.140625)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Subtracting loss instead of its gradient is not gradient descent. Squared gradients in Adam are coordinatewise. A zero gradient may identify a saddle point or maximum, not just a minimum.