M4.5 · LESSON 5 OF 6

Partial derivatives

Calculate slopes, use the chain rule and distinguish derivatives from partial derivatives.

THE IDEA

Many inputs, one at a time

Models have thousands of parameters. A loss such as f(x, y) = x² + 3xy depends on two inputs here. We cannot ask for "the" slope, because it depends on which direction you walk. The simplest directions are along the axes. A partial derivative changes one input and freezes the others.

∂f/∂x: treat y as a constant number. Then f is just x² + (3y)·x, whose derivative is 2x + 3y.
∂f/∂y: treat x as a constant. Then f is x² + (3x)·y, whose derivative is 3x.

At (2, 1): ∂f/∂x = 4 + 3 = 7 and ∂f/∂y = 6.

Slice the surface and read the slope

Picture f as a landscape and the colour map as a top view. A line of constant y through the point is a slice along x. The curve at the right is the height of the landscape along that slice, and the slope of its tangent is exactly ∂f/∂x. Choose the other slice and you read ∂f/∂y. The pair of numbers [∂f/∂x, ∂f/∂y] is the gradient.

Red: f large and positive. Blue: f negative. Click the map or use the sliders.

x 1y 1

slice along x (hold y = 1) slice along y (hold x = 1)

f(x, y) = x² + 3xy f(1, 1) = 4

∂f/∂x = 2x + 3y = 5 (numerical: 5.0000)

∂f/∂y = 3x = 3 (numerical: 3.0000)

gradient = [5, 3]

Click the map or use the sliders. Switch the slice between x and y and compare the green tangent with the numbers.

Two numerical partials of x² + 3xy at (2, 1)

f = lambda x, y: x ** 2 + 3 * x * y
h = 1e-5
print((f(2 + h, 1) - f(2 - h, 1)) / (2 * h))
print((f(2, 1 + h) - f(2, 1 - h)) / (2 * h))
# output:
7.0000000000902665
6.000000000039306

Predict first. For f(x, y) = x² + 3xy at (x, y) = (0, 5), what is ∂f/∂x?

Common mistakes

Differentiating y as if it changed when you take ∂/∂x. Freeze it: it behaves like the number 5 or −2.
Mixing up the symbol d and ∂. Use d for one-variable functions and ∂ when other inputs exist and are held fixed.
Thinking a zero partial means a flat point. At (3, −2) the x-slice is flat (∂f/∂x = 0) but the y-slice slopes at 9. Both partials must be 0 for a stationary point.

Chapter context

A derivative measures a local rate of change. It describes what a function does under a very small input change, rather than its average behavior over an arbitrary interval. This local sensitivity is what training algorithms use.

TRY IT YOURSELF

Part 1. Do it on the graph

Move the controls. Each task ticks from the real numbers on the screen and stays ticked once earned.

The surface is f(x, y) = x² + 3xy. Move the point and read the two slopes.

Red: f large and positive. Blue: f negative. Click the map or use the sliders.

x 1y 1

slice along x (hold y = 1) slice along y (hold x = 1)

f(x, y) = x² + 3xy f(1, 1) = 4

∂f/∂x = 2x + 3y = 5 (numerical: 5.0000)

∂f/∂y = 3x = 3 (numerical: 3.0000)

gradient = [5, 3]

To do: Put the point where the x-slope ∂f/∂x equals 7
To do: Find a point where the x-slice is flat (∂f/∂x = 0) while the y-slice is steep (∂f/∂y at least 6)
To do: Find the point where the y-slice is flat (∂f/∂y = 0) and the x-slope is exactly 6
To do: Find the point where both partial derivatives are 0 (the gradient is the zero vector)

Part 2. Do it in code

Write partial(f, point, i, h): the central-difference slope of f in coordinate i, holding all other coordinates fixed. f takes a list. Do not modify the list you were given.
Write gradient(f, point, h): the list of all partial derivatives.
Write check_gradient(f, grad, point, tol): True if the analytic gradient function grad(point) matches your numerical gradient within tol in every coordinate.

def partial(f, point, i, h=1e-5):
    # (f(point with coordinate i + h) - f(point with coordinate i - h)) / (2h)
    pass

def gradient(f, point, h=1e-5):
    # list of partial derivatives, one per coordinate
    pass

def check_gradient(f, grad, point, tol=1e-6):
    # True if grad(point) matches gradient(f, point) in every coordinate
    pass

f = lambda p: p[0] ** 2 + 3 * p[0] * p[1]
print(partial(f, [2, 1], 0))   # about 7

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

For x squared plus 3xy at x=2,y=1, find the x partial.

A5, because y is differentiated as if it were a function of x.B2x+3y=7.C6, because the derivative is taken with respect to y instead of x.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

f'(x)=\lim_{h\to0}\frac{f(x+h)-f(x)}h.

\frac{d}{dx}f(g(x))=f'(g(x))g'(x),\quad \frac{\partial}{\partial x}(x^2+3xy)=2x+3y.

For numerical checking, use $(f(x+h)-f(x-h))/(2h)$ and report a tolerance.

Worked example

Let f(x)=(3x+1)^2. At x=2, the inner value is 7, its derivative is 3, and the outer derivative is 14. Multiplying gives 42. A central difference with h=0.00001 gives approximately the same answer, within numerical error.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
f = lambda x: (3*x + 1)**2
x, h = 2, 1e-5
analytic = 6*(3*x + 1)
numeric = (f(x+h)-f(x-h))/(2*h)
assert analytic == 42
assert isclose(numeric, analytic, rel_tol=1e-7)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A function value is not its derivative. The chain rule multiplies sensitivities, not intermediate values. At a corner such as abs(x) at zero, a two-sided derivative does not exist.