M4.6 · LESSON 6 OF 6

Numerical checks

Calculate slopes, use the chain rule and distinguish derivatives from partial derivatives.

THE IDEA

Check a derivative with numbers

A formula you derived by hand can be wrong. A cheap test: compare it with a finite difference, a slope computed from nearby function values. Two versions:

Forward: (f(x + h) − f(x)) / h. Its error shrinks in proportion to h.
Central: (f(x + h) − f(x − h)) / (2h). The errors on the two sides cancel, so the error shrinks in proportion to h², far faster.

These are truncation errors: we cut the limit off at a finite h. It is natural to think a smaller h is always better. The chart below tests that with real double-precision numbers (the same your Python uses) for f = exp at x = 1, where the true derivative is e.

Smaller h is better, until it is not

h = 10^−k, k 4

forward central

f = exp, x = 1, true f′(1) = e = 2.7182818284590451

h = 0.0001

forward: 2.7184177470829241 error 1.36e-4

central: 2.7182818329896108 error 4.53e-9

f(x + h) = 2.7185536702337534

f(x) = 2.7182818284590451

forward numerator = 0.00027184, about 612133257640 units in the last place

central numerator = 0.00054366, about 1224205305014 units in the last place

A double can only change in whole units in the last place. When the numerator is only a handful of those units, the slope can take only a handful of values and the digits are noise.

Both axes are logarithmic. Move k to change h = 10^−k. The curves fall, bottom out and then climb again. Read the digits panel at k = 15 and 16.

The forward error falls as predicted down to about h = 1e-8, the central error down to about 1e-5. Beyond that the second source takes over: floating-point cancellation. Doubles keep about 16 significant digits. When h is tiny, f(x + h) and f(x − h) agree in almost every digit, and subtracting them wipes out the shared digits, leaving a few noisy ones that are then divided by a tiny h. At h = 1e-16 we even get x + h == x, so the numerator is exactly 0. Computed here: the forward error is 6.60e-9 at h = 1e-8 and 1.55e-6 at h = 1e-10, and the central error at h = 1e-15 is 1.68e-1.

Central difference of exp at 1: the estimate gets worse after a point

print(0.1 + 0.2 == 0.3, 0.1 + 0.2)
import math
x = 1.0
for h in [1e-5, 1e-10, 1e-15, 1e-16]:
    print(h, (math.exp(x + h) - math.exp(x - h)) / (2 * h))
# output:
False 0.30000000000000004
1e-05 2.718281828517632
1e-10 2.7182811557224795
1e-15 2.8865798640254066
1e-16 0.0

Predict first. With f = exp at x = 1, which step gives the better central-difference estimate of e?

What a numerical check can and cannot say

A check at a few points is evidence, not proof. And a kink can fool it: for |x| at 0 the central difference is 0.0, which looks like a perfectly good slope, while the forward difference is 1.0 and the backward difference is −1. Comparing forward and backward catches it.

Chapter context

A derivative measures a local rate of change. It describes what a function does under a very small input change, rather than its average behavior over an arbitrary interval. This local sensitivity is what training algorithms use.

TRY IT YOURSELF

Part 1. Do it on the graph

Move the controls. Each task ticks from the real numbers on the screen and stays ticked once earned.

Choose the method and h = 10^−k. The error is recomputed with real doubles each time.

h = 10^−k, k 1

forward central

f = exp, x = 1, true f′(1) = e = 2.7182818284590451

h = 0.1

forward: 2.8588419548738830 error 1.41e-1

central: 2.7228145639474177 error 4.53e-3

f(x + h) = 3.0041660239464334

f(x) = 2.7182818284590451

forward numerator = 0.28588, about 643753978134058 units in the last place

central numerator = 0.54456, about 1226246665559255 units in the last place

A double can only change in whole units in the last place. When the numerator is only a handful of those units, the slope can take only a handful of values and the digits are noise.

To do: Forward difference: choose h so the error is below 1e-7
To do: Central difference: choose h so the error is below 1e-9
To do: Choose an h where the central difference is at least 1000 times more accurate than the forward difference at that same h
To do: Break it: choose a central-difference h of 1e-15 or smaller and see an error above 0.1

Part 2. Do it in code

Write forward_diff(f, x, h) and central_diff(f, x, h).
Write has_kink(f, x, h=1e-6, tol=1e-3): True when the forward and backward differences disagree by more than tol, scaled by max(1, |forward|, |backward|) (a corner such as abs at 0).
Write check_derivative(f, df, xs, h=1e-5, tol=1e-6): True if the central difference matches df at every x in xs, with a tolerance that scales with the size of df(x), and False if f has a kink at any x.
Write best_h(f, df, x, hs): the h in hs whose central difference is closest to df(x).

def forward_diff(f, x, h):
    pass

def central_diff(f, x, h):
    pass

def has_kink(f, x, h=1e-6, tol=1e-3):
    # compare the forward difference (x to x + h) with the backward one (x - h to x);
    # scale tol by max(1, abs(forward), abs(backward)) so huge slopes do not look like corners
    pass

def check_derivative(f, df, xs, h=1e-5, tol=1e-6):
    # True if the central difference matches df(x) for every x; False at a kink.
    # Use a relative tolerance: |numeric - df(x)| <= tol * max(1, abs(df(x)))
    pass

def best_h(f, df, x, hs):
    # the h with the smallest central-difference error
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can a finite-difference check establish a symbolic proof for all x?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

f'(x)=\lim_{h\to0}\frac{f(x+h)-f(x)}h.

\frac{d}{dx}f(g(x))=f'(g(x))g'(x),\quad \frac{\partial}{\partial x}(x^2+3xy)=2x+3y.

For numerical checking, use $(f(x+h)-f(x-h))/(2h)$ and report a tolerance.

Worked example

Let f(x)=(3x+1)^2. At x=2, the inner value is 7, its derivative is 3, and the outer derivative is 14. Multiplying gives 42. A central difference with h=0.00001 gives approximately the same answer, within numerical error.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
f = lambda x: (3*x + 1)**2
x, h = 2, 1e-5
analytic = 6*(3*x + 1)
numeric = (f(x+h)-f(x-h))/(2*h)
assert analytic == 42
assert isclose(numeric, analytic, rel_tol=1e-7)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A function value is not its derivative. The chain rule multiplies sensitivities, not intermediate values. At a corner such as abs(x) at zero, a two-sided derivative does not exist.