D6.6 · LESSON 6 OF 6

Attention as a bridge

Trace a recurrent state and explain memory loss, gating, sequence-to-sequence models and attention.

THE IDEA

Let each step look back at all of the input

A plain encoder-decoder squeezes the whole input into one fixed-size vector. For long inputs that is a bottleneck: 12 words of 8 numbers each cannot all survive in 8 numbers. Attention removes it. The decoder keeps all encoder states and, at every step, computes a weighted average of them, with weights that depend on what it needs right now.

Score each position: the dot product of the decoder's query with that position's key.
Turn scores into weights with softmax (they sum to 1).
The context is the weighted sum of the values.

position	key	score = key · q	weight
the	[1, 0]	2	0.468
cat	[0, 1]	0	0.063
sat	[1, 1]	2	0.468

q[0]2q[1]0

mask out "cat" (cannot be attended)

weights = softmax(scores) (they sum to 1) context = Σ weight_i · value_i

Three positions with 2-number keys. Change the query to point at different words, and try the mask.

Predict first. Must a decoder attend only to the last encoder state?

The transformer chapters later compute these scores at scale, with scaling and many heads; here they are plain dot products.

Chapter context

A sequence model processes ordered observations. A recurrent state summarizes previous inputs, creating both a mechanism for memory and a long differentiation path whose sensitivity can decay or grow.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute attention weights by hand, aim a query at one word, and use a mask. Then implement softmax and attention in NumPy.

position	key	weight
the	[1, 0]	0.333
cat	[0, 1]	0.333
sat	[1, 1]	0.333

query q =

mask out "cat" (cannot be attended)

weights = softmax(scores) (they sum to 1) context = Σ weight_i · value_i

weight on "the" for q = [2, 0]

weight on "the" for q = [2, 0] with "cat" masked

The query boxes above are for the second task: make the weight on "cat" at least 0.7 (currently 0.333 without a mask).

To do: Keys: the = [1, 0], cat = [0, 1], sat = [1, 1]. For the query q = [2, 0], the scores are the dot products. Type the attention weight on "the" (3 decimals)
To do: Choose a query q so that the weight on "cat" is at least 0.7
To do: Mask out "cat" so it cannot be attended. With q = [2, 0], what is the weight on "the" now? Type it

Part 2. Do it in code

softmax(z): numerically stable softmax of a vector (subtract the maximum first). Entries equal to -inf get weight 0.
attention(q, K, V, mask=None): scores = K @ q (plain dot products, no scaling), optional mask (boolean array, True = allowed; masked scores become -inf), weights = softmax(scores), context = weights @ V. Return (weights, context). If every position is masked raise ValueError.
attend_all(Q, K, V): attention for every row of Q. Return (weights, contexts) with shapes (m, n) and (m, dv).

import numpy as np

def softmax(z):
    pass

def attention(q, K, V, mask=None):
    pass

def attend_all(Q, K, V):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Must a decoder attend only to the last encoder state?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A simple recurrence is $h_t=\phi(W_hh_{t-1}+W_xx_t+b)$ . In a scalar linear example $h_t=a h_{t-1}+x_t$ , sensitivity to the initial state is

\frac{\partial h_T}{\partial h_0}=a^T.

This illustrates gradient decay or growth; nonlinear multidimensional RNNs have Jacobian products instead.

Worked example

Set a=0.5 and h0=0. Input [1,0,0] produces states [1,0.5,0.25]. The initial impulse is still present but attenuated. Over ten subsequent multiplicative transitions, its factor is 0.5^10=0.0009765625.

Reference code

The following example uses the Python standard library and does not call a remote service.

state, states = 0.,[]
for x in [1,0,0]:
    state=.5*state+x
    states.append(state)
assert states == [1,.5,.25]
assert .5**10 == .0009765625

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A recurrent model shares weights through time. Longer hidden vectors do not automatically prevent gradient decay. Training history and generated inference history can differ.