D6.4 · LESSON 4 OF 6

LSTM and GRU gates

Trace a recurrent state and explain memory loss, gating, sequence-to-sequence models and attention.

THE IDEA

Gates decide what to keep, write and show

A gate is a number between 0 and 1 (from a sigmoid) that multiplies something: 0 blocks it, 1 lets it through. The LSTM keeps a separate memory called the cell state c and uses three gates around it.

c_new = f · c_old + i · g (forget some, write some) h_new = o · tanh(c_new) (expose some)

Because the cell update is an addition, with f near 1 the old memory passes through almost unchanged, and the gradient along it does not shrink. The cell state c and the output h are different quantities: c is the memory, h is what is shown to the next layer.

The old cell value is c_prev = 2 and the new candidate is g = 0.8.

LSTM cell, one step

forget f0.9input i0.5output o0.5

c = f·c_prev + i·g = 0.9·2 + 0.5·0.8 = 2.2 h = o·tanh(c) = 0.5·0.9757 = 0.4879

One LSTM cell step with a scalar memory. Move the gates and read the two results.

Predict first. Is an LSTM's cell state the same quantity as its output hidden state?

Chapter context

A sequence model processes ordered observations. A recurrent state summarizes previous inputs, creating both a mechanism for memory and a long differentiation path whose sensitivity can decay or grow.

TRY IT YOURSELF

Part 1. Do it hands-on

Set gates to produce three behaviours (keep, overwrite, hide) and compute a step by hand. Then write LSTM and GRU steps in NumPy.

Old cell value c_prev = 2, new candidate g = 0.8. Each row is a separate cell.

Row 1: keep the memory

forget f0.5input i0.5output o0.5

c = f·c_prev + i·g = 0.5·2 + 0.5·0.8 = 1.4 h = o·tanh(c) = 0.5·0.8854 = 0.4427

Row 2: overwrite the memory

forget f0.5input i0.5output o0.5

c = f·c_prev + i·g = 0.5·2 + 0.5·0.8 = 1.4 h = o·tanh(c) = 0.5·0.8854 = 0.4427

Row 3: remember silently

forget f0.5input i0.5output o0.5

c = f·c_prev + i·g = 0.5·2 + 0.5·0.8 = 1.4 h = o·tanh(c) = 0.5·0.8854 = 0.4427

By hand: f = 0.9, i = 0.5, o = 0.5 (g = 0.8, c_prev = 2)

c h (3 decimals)

To do: Gates row 1 (forget f, input i, output o): keep the old cell value c = 2 exactly
To do: Row 2: erase the old value and store the new candidate instead, c = 0.8
To do: Row 3: keep the cell value at 2 but expose nothing: the output h must be exactly 0
To do: With f = 0.9, i = 0.5, o = 0.5, g = 0.8, c_prev = 2: type c (exact) and h (3 decimals)

Part 2. Do it in code

lstm_step(x, h, c, P): one LSTM step with the standard equations. z = concatenate([x, h]); f = sigmoid(Wf@z + bf); i = sigmoid(Wi@z + bi); o = sigmoid(Wo@z + bo); g = tanh(Wg@z + bg); c_new = f*c + i*g; h_new = o*tanh(c_new). Return (h_new, c_new). P is a dict with keys Wf, Wi, Wo, Wg (shape (n, d+n)) and bf, bi, bo, bg (shape (n,)).
gru_step(x, h, P): z = sigmoid(Wz@[x,h] + bz); r = sigmoid(Wr@[x,h] + br); cand = tanh(Wh@[x, r*h] + bh); h_new = (1 - z)*h + z*cand. P has Wz, Wr, Wh (n, d+n) and bz, br, bh. (Other books swap the roles of z and 1-z; this exercise uses this convention.) Return h_new.
sigmoid is provided for you.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def lstm_step(x, h, c, P):
    pass

def gru_step(x, h, P):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Is an LSTM cell state the same quantity as its output hidden state?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A simple recurrence is $h_t=\phi(W_hh_{t-1}+W_xx_t+b)$ . In a scalar linear example $h_t=a h_{t-1}+x_t$ , sensitivity to the initial state is

\frac{\partial h_T}{\partial h_0}=a^T.

This illustrates gradient decay or growth; nonlinear multidimensional RNNs have Jacobian products instead.

Worked example

Set a=0.5 and h0=0. Input [1,0,0] produces states [1,0.5,0.25]. The initial impulse is still present but attenuated. Over ten subsequent multiplicative transitions, its factor is 0.5^10=0.0009765625.

Reference code

The following example uses the Python standard library and does not call a remote service.

state, states = 0.,[]
for x in [1,0,0]:
    state=.5*state+x
    states.append(state)
assert states == [1,.5,.25]
assert .5**10 == .0009765625

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A recurrent model shares weights through time. Longer hidden vectors do not automatically prevent gradient decay. Training history and generated inference history can differ.