D6.3 · LESSON 3 OF 6

Vanishing and exploding memory

Trace a recurrent state and explain memory loss, gating, sequence-to-sequence models and attention.

THE IDEA

Long products shrink or blow up

A long product of the same factor does one of three things: if the factor is below 1 the result decays toward zero (vanishing), if it is above 1 it grows enormously (exploding), and only a factor close to 1 keeps it steady. For matrices the same story is told by the largest scale the matrix stretches vectors by.

factor per step0.5steps10

0.5^10 = 9.766e-4 (vanished)

Bars show the size of the signal after each step (log-compressed, capped for display).

gradient length50

gradient [30, 40] has length 50 clipped to length 5: [3, 4] (direction unchanged)

Pick a factor and a number of steps. Below: what gradient clipping does to a huge gradient.

The worked example: with a = 0.5, an impulse [1, 0, 0] gives states [1, 0.5, 0.25]. After ten further multiplications its share is 0.5¹⁰ = 0.0009765625, about a thousandth.

Predict first. What is ten repeated factors of 0.5?

Chapter context

A sequence model processes ordered observations. A recurrent state summarizes previous inputs, creating both a mechanism for memory and a long differentiation path whose sensitivity can decay or grow.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute vanishing and exploding products and clip a gradient by hand, then code decay, clipping and matrix growth.

factor per step0.5steps10

0.5^10 = 9.766e-4 (vanished)

Bars show the size of the signal after each step (log-compressed, capped for display).

gradient length50

gradient [30, 40] has length 50 clipped to length 5: [3, 4] (direction unchanged)

0.5 repeated 10 times

1.5 repeated 10 times (2 decimals)

steps for 0.9 per step to drop below 0.01

clip [30, 40] to length 5:

To do: Multiply 0.5 by itself ten times (0.5¹⁰). Type the value (a fraction like 1/1024 is fine)
To do: Now 1.5 ten times (1.5¹⁰), rounded to two decimals
To do: A signal loses 10% per step (factor 0.9). After how many steps is it first below 0.01?
To do: Gradient clipping: the gradient is [30, 40] (norm 50) and the limit is 5. Type the clipped gradient

Part 2. Do it in code

decay_factor(a, n): a repeated n times, a**n. Raise ValueError for n < 0.
steps_until_below(a, threshold): the number of multiplications by a (starting from 1) until the value is STRICTLY below the threshold. Requires 0 < a < 1 and 0 < threshold < 1, otherwise ValueError. Use a loop (not a formula) so ties are handled correctly.
clip_by_norm(g, max_norm): if the vector's length (L2 norm) is larger than max_norm, scale it to have exactly that length; otherwise return an unchanged copy. max_norm must be positive (ValueError).
growth_norms(W, v, n): the lengths of v, W@v, W@W@v, ... as a list of n + 1 numbers.

import numpy as np

def decay_factor(a, n):
    pass

def steps_until_below(a, threshold):
    pass

def clip_by_norm(g, max_norm):
    pass

def growth_norms(W, v, n):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

What is ten repeated factors of .5?

A5, because ten factors of 0.5 add up to 5.B0.05, because ten factors of 0.5 add up to 5 tenths.C1/1024 (about 0.001), because 0.5¹⁰ = 1/2¹⁰.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A simple recurrence is $h_t=\phi(W_hh_{t-1}+W_xx_t+b)$ . In a scalar linear example $h_t=a h_{t-1}+x_t$ , sensitivity to the initial state is

\frac{\partial h_T}{\partial h_0}=a^T.

This illustrates gradient decay or growth; nonlinear multidimensional RNNs have Jacobian products instead.

Worked example

Set a=0.5 and h0=0. Input [1,0,0] produces states [1,0.5,0.25]. The initial impulse is still present but attenuated. Over ten subsequent multiplicative transitions, its factor is 0.5^10=0.0009765625.

Reference code

The following example uses the Python standard library and does not call a remote service.

state, states = 0.,[]
for x in [1,0,0]:
    state=.5*state+x
    states.append(state)
assert states == [1,.5,.25]
assert .5**10 == .0009765625

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A recurrent model shares weights through time. Longer hidden vectors do not automatically prevent gradient decay. Training history and generated inference history can differ.