M4.4 · LESSON 4 OF 6

The chain rule

Calculate slopes, use the chain rule and distinguish derivatives from partial derivatives.

THE IDEA

A chain of machines

Many functions are built by feeding one function into another: first compute u = 3x + 1, then square it: y = u². The pair is a composition. Two questions: what is the slope of y with respect to x, and how do we get it without expanding everything?

Think of slopes as magnification. A tiny step in x becomes a step 3 times as large in u, because the inner machine has slope 3. A tiny step in u becomes a step 2u times as large in y, because the outer machine has slope 2u. Magnifications multiply: y reacts to x by 2u · 3. That is the chain rule: dy/dx = (dy/du) · (du/dx).

composition

x 2

x = 2

→

inner
u = 3x + 1
slope 3

→

u = 7

→

outer
y = u²
slope 2u = 14

→

y = 49

dy/dx = (dy/du) · (du/dx) = 14 · 3 = 42

check: central difference of the whole machine = 42.0000

wrong: outer slope evaluated at x instead of u gives 12

Each stage magnifies a tiny step by its own slope. The magnifications multiply, so the slopes multiply. Bar lengths are drawn to scale, capped at the edge of the picture.

Choose x and a composition. The bars at the bottom show a tiny step growing at each stage, drawn to scale.

The worked example, with the right and wrong evaluation points

For y = (3x + 1)² at x = 2: the inner value is u = 7, the inner slope is 3, the outer slope 2u is 14, and 14 · 3 = 42. Python agrees:

Numerical slope, the mistake and the chain rule at x = 2

f = lambda x: (3 * x + 1) ** 2
h = 1e-5
print((f(2 + h) - f(2 - h)) / (2 * h))
print(2 * 2 * 3)
print(2 * (3 * 2 + 1) * 3)
# output:
42.000000000896875
12
42

The mistake used 2x = 4 as the outer slope and got 12. The outer derivative must be evaluated at the inner output u = 3x + 1 = 7, not at x.

Predict first. What is the derivative of (3x + 1)² at x = 0?

Why this matters for learning

A neural network is a long chain of functions. The loss L = (w·x + b − t)² is a chain: inner r = w·x + b − t, outer r². The slope with respect to w is 2r · x and with respect to b is 2r · 1. Backpropagation is the chain rule applied systematically along such chains. You write exactly these functions in the practice.

Chapter context

A derivative measures a local rate of change. It describes what a function does under a very small input change, rather than its average behavior over an arbitrary interval. This local sensitivity is what training algorithms use.

TRY IT YOURSELF

Part 1. Do it on the graph

Move the controls. Each task ticks from the real numbers on the screen and stays ticked once earned.

The machine is y = (3x + 1)². Move x and read dy/dx.

x 0

x = 0

→

inner
u = 3x + 1
slope 3

→

u = 1

→

outer
y = u²
slope 2u = 2

→

y = 1

dy/dx = (dy/du) · (du/dx) = 2 · 3 = 6

check: central difference of the whole machine = 6.0000

wrong: outer slope evaluated at x instead of u gives 0

Each stage magnifies a tiny step by its own slope. The magnifications multiply, so the slopes multiply. Bar lengths are drawn to scale, capped at the edge of the picture.

To do: Find the x where dy/dx = 42 (the worked example of this chapter)
To do: Find the x where the curve is falling with slope −30
To do: Find the x where the common mistake 2x · 3 (outer derivative evaluated at x, not at u) matches the true slope

Part 2. Do it in code

Write chain(fp, g, gp, x): the derivative of f(g(x)) given fp (the derivative function of f), g and gp (the derivative of g). Remember where fp must be evaluated.
Write sq_err_grad(w, b, x, t): for the loss L = (w·x + b − t)², return the pair (dL/dw, dL/db). The inner function is w·x + b − t.
Write sigmoid_sq_deriv(x, a, b): the derivative of (σ(a·x + b))² with respect to x, where σ(z) = 1 / (1 + e^(−z)) and σ′ = σ(1 − σ). That is a three-layer chain.

import math

def chain(fp, g, gp, x):
    # derivative of f(g(x)): outer derivative at g(x), times inner derivative at x
    pass

def sq_err_grad(w, b, x, t):
    # L = (w*x + b - t)**2 ; return (dL/dw, dL/db)
    pass

def sigmoid_sq_deriv(x, a, b):
    # d/dx of sigmoid(a*x + b)**2
    pass

print(chain(lambda u: 2 * u, lambda x: 3 * x + 1, lambda x: 3, 2))  # should print 42

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Differentiate (3x+1) squared.

A6x, because the outer derivative is evaluated at x instead of at 3x+1.B2(3x+1)*3 = 6(3x+1).C2(3x+1), because the inner derivative 3 is left out.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

f'(x)=\lim_{h\to0}\frac{f(x+h)-f(x)}h.

\frac{d}{dx}f(g(x))=f'(g(x))g'(x),\quad \frac{\partial}{\partial x}(x^2+3xy)=2x+3y.

For numerical checking, use $(f(x+h)-f(x-h))/(2h)$ and report a tolerance.

Worked example

Let f(x)=(3x+1)^2. At x=2, the inner value is 7, its derivative is 3, and the outer derivative is 14. Multiplying gives 42. A central difference with h=0.00001 gives approximately the same answer, within numerical error.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
f = lambda x: (3*x + 1)**2
x, h = 2, 1e-5
analytic = 6*(3*x + 1)
numeric = (f(x+h)-f(x-h))/(2*h)
assert analytic == 42
assert isclose(numeric, analytic, rel_tol=1e-7)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A function value is not its derivative. The chain rule multiplies sensitivities, not intermediate values. At a corner such as abs(x) at zero, a two-sided derivative does not exist.