D2.2 · LESSON 2 OF 6

Local derivatives

Apply the chain rule through a graph and implement the core of a scalar reverse-mode autodiff engine.

THE IDEA

Every operation knows its own slope

A local derivative answers a tiny question about one node: if I nudge one of this node's inputs, how much does this node's output move? Each operation has a simple rule, and the rule often uses the values from the forward pass.

operation	output	local derivative(s)
add	a + b	1 and 1
multiply	a · b	b with respect to a, a with respect to b
square	u²	2u
tanh	t = tanh(z)	1 − t²
relu	max(0, z)	1 if z > 0, otherwise 0

Predict first. For p = a·b, what is dp/da?

Local derivative times upstream gradient

One node never looks beyond its neighbours. It takes the gradient that arrives from above (how much the loss cares about this node's output), multiplies it by each local derivative, and hands the products to its inputs. That single move, repeated, is the chain rule.

0 of 3 operation nodes processed

start: ∂L/∂L = 1, because the loss changes one-for-one with itself.

Take a=2, b=3, c=1, so u = 7 and L = 49. Press the button to send the gradient back one node at a time and read the arithmetic.

Chapter context

Backpropagation organizes repeated applications of the chain rule. The graph stores how values depend on earlier values. Reverse traversal accumulates how the final loss changes with each intermediate quantity.

TRY IT YOURSELF

Part 1. Do it hands-on

Read the forward values off the graph and fill in each local derivative.

forward values: a = 2, b = 3, p = a·b = 6, c = 1, u = p + c = 7, L = u² = 49

edge	local derivative
∂L/∂u (square)
∂u/∂p (add)
∂u/∂c (add)
∂p/∂a (multiply)
∂p/∂b (multiply)

relu′ at −2: relu′ at 3:

To do: Square node: type the local derivative of L with respect to u (L = u²) at the value u has now
To do: Add node: type the local derivative of u with respect to p and with respect to c
To do: Multiply node: type the local derivative of p with respect to a, and with respect to b
To do: relu(x) = max(0, x): what is its local derivative when the input is −2, and when it is 3?

Part 2. Do it in code

Write local_grads(op, args): a tuple with the derivative of the operation's output with respect to each argument, evaluated at args. Operations: add, sub, mul, div, sq, tanh, relu, neg (use the OPS table to get the output when you need it).
relu clamps: where the input is 0 or negative the local derivative is 0, not 1. Division gives (1/b, −a/b²).
Write contribution(upstream, local): the chain-rule product that travels along one edge. Then write edge_report(op, args, upstream) that returns the list of contributions, one per argument.

import math

OPS = {
    "add": lambda a, b: a + b,
    "sub": lambda a, b: a - b,
    "mul": lambda a, b: a * b,
    "div": lambda a, b: a / b,
    "sq": lambda a: a * a,
    "tanh": lambda a: math.tanh(a),
    "relu": lambda a: a if a > 0 else 0.0,
    "neg": lambda a: -a,
}

def local_grads(op, args):
    # one number per argument: d(output)/d(argument), evaluated at args
    pass

def contribution(upstream, local):
    pass

def edge_report(op, args, upstream):
    pass

print(local_grads("mul", (2.0, 3.0)))   # should be (3.0, 2.0)

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

For p=a*b, what is dp/da?

A1, because multiplying behaves like adding for derivatives.Bb, because nudging a by a small amount changes p = a·b by b times that amount.Ca, because the derivative with respect to a is the value of a.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

If $u=ab+c$ and $L=u^2$ , then

\frac{\partial L}{\partial a}=2ub,\quad \frac{\partial L}{\partial b}=2ua,\quad \frac{\partial L}{\partial c}=2u.

At a shared node x, add all downstream contributions: $\partial L/\partial x=\sum_j(\partial L/\partial u_j)(\partial u_j/\partial x)$ .

Worked example

Take a=2,b=3,c=1. Then u=7 and L=49. The upstream gradient into u is 14, giving dL/da=14×3=42, dL/db=14×2=28 and dL/dc=14. These are sensitivities, not new parameter values; an optimizer still needs a step size.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
a,b,c = 2.,3.,1.
u = a*b+c
assert u*u == 49
assert (2*u*b,2*u*a,2*u) == (42,28,14)
h = 1e-5
f = lambda x: (x*b+c)**2
assert isclose((f(a+h)-f(a-h))/(2*h), 42, rel_tol=1e-7)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Forward values and gradients are different objects. Shared branches require addition of gradient contributions. Backpropagation computes derivatives; the optimizer changes parameters.