Gradient checks
Apply the chain rule through a graph and implement the core of a scalar reverse-mode autodiff engine.
THE IDEA
Trust, but check numerically
Backpropagation bugs are quiet: the code runs, the loss even goes down a little, but a derivative rule is wrong. A gradient check catches this by comparing each analytic gradient with an estimate from the function itself, the central difference:
numeric ≈ ( L(w + h) − L(w − h) ) / (2h)
relative error = |analytic − numeric| / (|analytic| + |numeric|)Try it on a tanh neuron. With a correct engine every parameter passes. Plant a wrong tanh rule and the parameters that flow through it fail, while the ones that do not pass through it still pass. That pattern tells you where to look.
| param | backprop | numeric (central diff) | relative error | verdict |
|---|---|---|---|---|
| w | −3.085427 | −3.085427 | 1.3e-11 | pass |
| x | 0.462814 | 0.462814 | 2.6e-12 | pass |
| b | −1.542714 | −1.542714 | 5.2e-12 | pass |
| y | 1.605249 | 1.605249 | 3.0e-12 | pass |
h = 10^-5. A very large h measures the average slope over a wide step; a tiny h drowns in rounding error. Around 1e-5 the correct engine passes.
Predict first. Is demanding exact floating-point equality a good gradient check?
Chapter context
Backpropagation organizes repeated applications of the chain rule. The graph stores how values depend on earlier values. Reverse traversal accumulates how the final loss changes with each intermediate quantity.
TRY IT YOURSELF
Part 1. Do it hands-on
Two engines each have one wrong local rule. Use the check to find which parameters fail, locate the faulty node, and choose the correct rule.
| param | backprop | numeric (central diff) | relative error | verdict |
|---|---|---|---|---|
| w | −2.576826 | −3.085427 | 9.0e-2 | FAIL |
| x | 0.386524 | 0.462814 | 9.0e-2 | FAIL |
| b | −1.288413 | −1.542714 | 9.0e-2 | FAIL |
| y | 1.605249 | 1.605249 | 3.0e-12 | pass |
| param | backprop | numeric (central diff) | relative error | verdict |
|---|---|---|---|---|
| w | 0.462814 | −3.085427 | 1.0e+0 | FAIL |
| x | −3.085427 | 0.462814 | 1.0e+0 | FAIL |
| b | −1.542714 | −1.542714 | 5.2e-12 | pass |
| y | 1.605249 | 1.605249 | 3.0e-12 | pass |
- To do: Engine A: run the gradient check and tick every parameter whose gradient fails
- To do: Engine A: which node has the wrong local derivative?
- To do: Engine A: choose the correct rule for that node so that every parameter passes
- To do: Engine B: tick every parameter whose gradient fails
- To do: Engine B: which node has the wrong local derivative?
- To do: Engine B: choose the correct rule so that every parameter passes
Part 2. Do it in code
- f takes a list of numbers and returns one number. Write central_diff(f, x, i, h=1e-5): the central-difference derivative of f with respect to x[i]. It must not change the caller's list.
- Write rel_error(a, b) = |a − b| / (|a| + |b|), and 0.0 when both are 0.
- Write check_grads(f, analytic, x, h=1e-5, tol=1e-6): analytic is the list of backprop gradients. Return a list of booleans: True where the relative error between analytic[i] and the central difference is below tol.
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Does exact floating point equality make a good finite-difference check?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
If and , then
At a shared node x, add all downstream contributions: .
Worked example
Take a=2,b=3,c=1. Then u=7 and L=49. The upstream gradient into u is 14, giving dL/da=14×3=42, dL/db=14×2=28 and dL/dc=14. These are sensitivities, not new parameter values; an optimizer still needs a step size.
Reference code
The following example uses the Python standard library and does not call a remote service.
from math import isclose
a,b,c = 2.,3.,1.
u = a*b+c
assert u*u == 49
assert (2*u*b,2*u*a,2*u) == (42,28,14)
h = 1e-5
f = lambda x: (x*b+c)**2
assert isclose((f(a+h)-f(a-h))/(2*h), 42, rel_tol=1e-7)
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
Forward values and gradients are different objects. Shared branches require addition of gradient contributions. Backpropagation computes derivatives; the optimizer changes parameters.