D6.5 · LESSON 5 OF 6

Sequence-to-sequence

Trace a recurrent state and explain memory loss, gating, sequence-to-sequence models and attention.

THE IDEA

Encoder, decoder and the training shortcut

For tasks like translation both the input and the output are sequences, and they can differ in length. A sequence-to-sequence model has an encoder that reads the input into a representation and a decoder that writes the output one token at a time, each token conditioned on the ones already written.

During training we know the correct answer, so the decoder is given the previous reference token at every step instead of its own (possibly wrong) previous output. That is teacher forcing: the inputs are the target sequence shifted right by one, starting with a start symbol. The input at step t must never be the label at step t, or the model could just copy the answer.

Target: le chat dort </s>. This toy decoder predicts the next word from only the word it was just given, and it has one learned mistake: after "le" it says "chien".

Step 1 of 4

step 1 input fed in : <s> (the reference word) prediction : le reference : le ✓

The same imperfect decoder, trained-style versus used-style. Step through both modes.

Predict first. With teacher forcing, what does the decoder receive as input at step 3 of the target "le chat dort"?

Python

inputs = [BOS] + target          # what the decoder sees
labels = target + [EOS]          # what it must predict at each step

Chapter context

A sequence model processes ordered observations. A recurrent state summarizes previous inputs, creating both a mechanism for memory and a long differentiation path whose sensitivity can decay or grow.

TRY IT YOURSELF

Part 1. Do it hands-on

Build the teacher-forcing inputs, predict what happens at inference, then write the training pairs and a greedy decoder.

decoder inputs (teacher forcing):

input at step 3 at inference, after predictions le, chien

numbers attention can read (12 words × 8)

Target: le chat dort </s>. This toy decoder predicts the next word from only the word it was just given, and it has one learned mistake: after "le" it says "chien".

Step 1 of 4

step 1 input fed in : <s> (the reference word) prediction : le reference : le ✓

To do: Target: le chat dort </s>. Teacher forcing: type the four decoder INPUTS (one per step)
To do: At inference there is no reference. The decoder predicted le, then chien. What token is fed in at step 3? Type it
To do: A source of 12 words is encoded with 8 numbers per word. A fixed-size bottleneck keeps one 8-number vector. How many numbers can attention look at? Type it

Part 2. Do it in code

shift_right(tokens, bos): return a new list with bos in front and the last token dropped.
teacher_forcing_pair(target, bos, eos): return (inputs, labels) for one training sequence. inputs = [bos] + target, labels = target + [eos]. At step t the decoder sees inputs[t] and must predict labels[t].
greedy_decode(next_token, bos, eos, max_len): start with the input bos; repeatedly call next_token(previous) to get the next token, feeding the model's OWN previous output. Stop after producing eos (include it) or after max_len tokens. Return the generated list.
inference_inputs(next_token, bos, steps): the inputs the decoder really receives at inference: [bos, p1, p2, ...] where p_k are its own predictions. Return exactly `steps` inputs.

def shift_right(tokens, bos):
    pass

def teacher_forcing_pair(target, bos, eos):
    pass

def greedy_decode(next_token, bos, eos, max_len):
    pass

def inference_inputs(next_token, bos, steps):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does teacher forcing expose the next reference token as the current prediction input?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A simple recurrence is $h_t=\phi(W_hh_{t-1}+W_xx_t+b)$ . In a scalar linear example $h_t=a h_{t-1}+x_t$ , sensitivity to the initial state is

\frac{\partial h_T}{\partial h_0}=a^T.

This illustrates gradient decay or growth; nonlinear multidimensional RNNs have Jacobian products instead.

Worked example

Set a=0.5 and h0=0. Input [1,0,0] produces states [1,0.5,0.25]. The initial impulse is still present but attenuated. Over ten subsequent multiplicative transitions, its factor is 0.5^10=0.0009765625.

Reference code

The following example uses the Python standard library and does not call a remote service.

state, states = 0.,[]
for x in [1,0,0]:
    state=.5*state+x
    states.append(state)
assert states == [1,.5,.25]
assert .5**10 == .0009765625

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A recurrent model shares weights through time. Longer hidden vectors do not automatically prevent gradient decay. Training history and generated inference history can differ.