D6.1 · LESSON 1 OF 6

Recurrent state

Trace a recurrent state and explain memory loss, gating, sequence-to-sequence models and attention.

THE IDEA

A network that reads one step at a time

Text, audio and sensor streams are sequences whose length varies. A recurrent network reads them one element at a time and carries a state h that summarises everything it has seen so far. At every step the same small function updates it: new state = f(old state, current input).

h_t = tanh(Wx · x_t + Wh · h_(t−1) + b)

The important detail is that Wx, Wh and b are the same at every step. The state changes; the weights do not. That is why a recurrent network has the same number of parameters whether it reads 3 steps or 3,000.

recurrent weight a0.5

Click an input to toggle it between 0 and 1.

step t	1	2	3	4	5	6
input x_t
state h_t	1	0.5	0.25	0.125	0.063	0.031

h_t = a · h_(t−1) + x_t (the SAME a at every step) the first input is still in h_6: 0.5^5 = 0.0313 of it remains

The simplest case, one number and no nonlinearity: h_t = a·h_(t−1) + x_t. Send an impulse in step 1 and watch it fade.

Predict first. Does a ten-step RNN normally have ten unrelated copies of its recurrent weights?

NumPy: the whole forward pass

h = np.zeros(n)
for x in xs:                       # one row of xs per step
    h = np.tanh(Wx @ x + Wh @ h + b)   # same Wx, Wh, b each time

Chapter context

A sequence model processes ordered observations. A recurrent state summarizes previous inputs, creating both a mechanism for memory and a long differentiation path whose sensitivity can decay or grow.

TRY IT YOURSELF

Part 1. Do it hands-on

Trace a state by hand, count parameters, then write the forward pass of an RNN in NumPy.

states for inputs [1, 0, 0], a = 0.5:

parameters (3 inputs, 4 hidden)

parameters after running it for 10 steps

recurrent weight a0.5

Click an input to toggle it between 0 and 1.

step t	1	2	3	4	5	6
input x_t
state h_t	1	0.5	0.25	0.125	0.063	0.031

h_t = a · h_(t−1) + x_t (the SAME a at every step) the first input is still in h_6: 0.5^5 = 0.0313 of it remains

To do: Trace h_t = 0.5·h_(t−1) + x_t from h_0 = 0 with inputs [1, 0, 0]. Type the three states
To do: An RNN with 3 inputs and 4 hidden units has weights Wx (4×3), Wh (4×4) and a bias (4). How many parameters?
To do: The same network is run over a sequence of 10 steps. How many parameters does it have now?

Part 2. Do it in code

rnn_forward(xs, Wx, Wh, b, h0=None): xs has shape (T, d), Wx (n, d), Wh (n, n), b (n,). For each step compute h = tanh(Wx @ x + Wh @ h + b), starting from h0 (zeros if None). Return all T states as an array of shape (T, n). Raise ValueError if the width of xs does not match Wx.
rnn_param_count(d, n): number of parameters of the cell (Wx, Wh and b).
rnn_final_states(X, Wx, Wh, b): X has shape (B, T, d). Return an array (B, n) with the last state of every sequence.

import numpy as np

def rnn_forward(xs, Wx, Wh, b, h0=None):
    pass

def rnn_param_count(d, n):
    pass

def rnn_final_states(X, Wx, Wh, b):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does a ten-step RNN normally have ten unrelated copies of its recurrent weights?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A simple recurrence is $h_t=\phi(W_hh_{t-1}+W_xx_t+b)$ . In a scalar linear example $h_t=a h_{t-1}+x_t$ , sensitivity to the initial state is

\frac{\partial h_T}{\partial h_0}=a^T.

This illustrates gradient decay or growth; nonlinear multidimensional RNNs have Jacobian products instead.

Worked example

Set a=0.5 and h0=0. Input [1,0,0] produces states [1,0.5,0.25]. The initial impulse is still present but attenuated. Over ten subsequent multiplicative transitions, its factor is 0.5^10=0.0009765625.

Reference code

The following example uses the Python standard library and does not call a remote service.

state, states = 0.,[]
for x in [1,0,0]:
    state=.5*state+x
    states.append(state)
assert states == [1,.5,.25]
assert .5**10 == .0009765625

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A recurrent model shares weights through time. Longer hidden vectors do not automatically prevent gradient decay. Training history and generated inference history can differ.