L4.5 · LESSON 5 OF 6

Weighted values

Calculate query-key scores, scaled softmax weights and the weighted value output for one attention head.

THE IDEA

Mixing the values

The last step multiplies the weight row by the value matrix: the output for a query is the sum of the value vectors, each scaled by its weight. Because the weights are non-negative and add to 1, the result is a weighted average, so it lies between the values being mixed.

Worked example: q = [1, 0], K = [[1,0],[0,1]], V = [[2,0],[0,2]], d_k = 2. Scores [1, 0] become [0.7071, 0], the weights are [0.669762, 0.330238], and the output is [1.339523, 0.660477].

weight on value 10.25

weights [0.25, 0.75] on scalar values [2, 6] output = 0.25·2 + 0.75·6 = 5 (always between 2 and 6)

chapter example	value
q = [1, 0], K = [[1,0],[0,1]], d_k = 2	scores [1, 0] → scaled [0.7071, 0]
weights	[0.669762, 0.330238]
V = [[2,0],[0,2]] → output	[1.339523, 0.660477]

The output has the width of a value vector, not the length of the sequence: weights (n_q × n_k) @ V (n_k × d_v) = (n_q × d_v).

Slide the weight between two scalar values, and see the chapter example.

Predict first. Weights [0.25, 0.75] combine scalar values 2 and 6 into what?

NumPy: the whole head

s = Q @ K.T / np.sqrt(d_k)
w = softmax(s, axis=-1)            # one distribution per query
out = w @ V                        # (n_q, d_v)

Chapter context

Attention computes a data-dependent mixture of value vectors. A query scores candidate keys, a normalization turns scores into weights, and those weights combine values. Keys decide how to look; values supply what is gathered.

TRY IT YOURSELF

Part 1. Do it hands-on

Mix values by hand, then implement a complete single attention head and a validity check for its weights.

weight on value 10.25

weights [0.25, 0.75] on scalar values [2, 6] output = 0.25·2 + 0.75·6 = 5 (always between 2 and 6)

chapter example	value
q = [1, 0], K = [[1,0],[0,1]], d_k = 2	scores [1, 0] → scaled [0.7071, 0]
weights	[0.669762, 0.330238]
V = [[2,0],[0,2]] → output	[1.339523, 0.660477]

The output has the width of a value vector, not the length of the sequence: weights (n_q × n_k) @ V (n_k × d_v) = (n_q × d_v).

0.25·2 + 0.75·6 first output component (3 decimals) output columns for V of width 5

To do: Weights [0.25, 0.75] combine the scalar values 2 and 6. The result?
To do: Chapter example: weights [0.669762, 0.330238] and V = [[2, 0], [0, 2]]. Type the first component of the output (3 decimals)
To do: 3 queries, 3 keys, values of size 5. The weight matrix is 3×3; V is 3×5. How many columns does the output have?

Part 2. Do it in code

weighted_values(W, V): the matrix product W @ V. Raise ValueError if W has a different number of columns than V has rows.
attention(Q, K, V): single-head scaled dot-product attention without a mask. Return (weights, output) where weights = softmax(Q @ K.T / sqrt(d_k)) taken row by row and output = weights @ V.
attention_weights_valid(W): True if all weights are between 0 and 1 and every row sums to 1.

import numpy as np

def weighted_values(W, V):
    pass

def attention(Q, K, V):
    pass

def attention_weights_valid(W):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Weights [.25,.75] combine scalar values 2 and 6 into what?

AEight, because the weights add the two values.BFour, because the average ignores the weights.CFive: 0.25·2 + 0.75·6 = 0.5 + 4.5.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

Q\in\mathbb R^{n_q\times d_k},\ K\in\mathbb R^{n_k\times d_k},\ V\in\mathbb R^{n_k\times d_v}.

S=QK^T/\sqrt{d_k}+M,\quad A=\operatorname{softmax}_{rows}(S),\quad O=AV.

M is zero at allowed positions and conceptually negative infinity at excluded positions. Every query needs a defined policy if all its keys are masked.

Worked example

Use q=[1,0], K=[[1,0],[0,1]], V=[[2,0],[0,2]], and d_k=2. Dot scores [1,0] become [1/sqrt(2),0]. Exponentiate and normalize to approximately [0.669762,0.330238]. The output is [2×0.669762,2×0.330238]≈[1.339523,0.660477]. If only the first key is causally allowed, the output becomes [2,0].

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sqrt,exp,isclose
q=[1,0]; keys=[[1,0],[0,1]]; values=[[2,0],[0,2]]
scores=[sum(a*b for a,b in zip(q,k))/sqrt(2) for k in keys]
weights=[exp(s-max(scores)) for s in scores]
weights=[w/sum(weights) for w in weights]
out=[sum(weights[i]*values[i][j] for i in range(2)) for j in range(2)]
assert isclose(sum(weights),1)
assert isclose(weights[0],.6697615493266569)
assert isclose(out[0],1.3395230986533138)
assert isclose(out[1],.6604769013466862)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Attention scores are not weights before normalization. Causal masking happens before softmax. Attention weights alone are not a full causal explanation of a model decision.