L4.2 · LESSON 2 OF 6

Dot-product scores

Calculate query-key scores, scaled softmax weights and the weighted value output for one attention head.

THE IDEA

How well does a key match the query?

The match between a query and a key is their dot product: large and positive when they point the same way, zero when perpendicular, negative when they point in opposite directions. One query gets one score per allowed key. At this stage the scores are just real numbers, not probabilities.

q[0]1q[1]0

key	vector	score = q · key
1	[1, 0]	1
2	[0, 1]	0
3	[-1, 0.5]	-1

Scores are plain real numbers; a negative one means "pointing away". They become probabilities only after the softmax.

Move the query. The longer the overlap with a key's direction, the higher its score.

Predict first. For q = [1, 0] and keys [1, 0] and [0, 1], what are the scores?

NumPy

scores = K @ q          # one score per key
S = Q @ K.T             # all queries at once: (n_q, n_k)

Chapter context

Attention computes a data-dependent mixture of value vectors. A query scores candidate keys, a normalization turns scores into weights, and those weights combine values. Keys decide how to look; values supply what is gathered.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute scores by hand and aim a query at one key. Then write the score functions in Python.

key	vector	score = q · key
1	[1, 0]	1
2	[0, 1]	0
3	[-1, 0.5]	-1

Scores are plain real numbers; a negative one means "pointing away". They become probabilities only after the softmax.

scores for q=[1,0], keys [1,0],[0,1]:

scores for q=[1,−1], keys [1,0],[−1,1]:

your query q (for the third task) =

To do: q = [1, 0] against keys [1, 0] and [0, 1]. Type the two scores (dot products)
To do: q = [1, −1] against keys [1, 0] and [−1, 1]. Type the two scores (one is negative)
To do: Choose a query q (two numbers) so the key [0, 1] scores HIGHER than the key [1, 0]

Part 2. Do it in code

scores(q, K): the dot product of the query q with every row of K (a vector with one score per key).
score_matrix(Q, K): all query-key dot products, shape (n_q, n_k).
best_key(q, K): the index of the key with the highest score (the first one on ties).
ranked_keys(q, K): the key indices ordered from highest to lowest score.

import numpy as np

def scores(q, K):
    pass

def score_matrix(Q, K):
    pass

def best_key(q, K):
    pass

def ranked_keys(q, K):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

For q=[1,0], keys [1,0] and [0,1], what are the scores?

A[1, 1], because both keys overlap with the query.B[1, 0]: 1·1 + 0·0 = 1 for the first key and 1·0 + 0·1 = 0 for the second.C[0, 1], because the scores follow the order of the dimensions.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

Q\in\mathbb R^{n_q\times d_k},\ K\in\mathbb R^{n_k\times d_k},\ V\in\mathbb R^{n_k\times d_v}.

S=QK^T/\sqrt{d_k}+M,\quad A=\operatorname{softmax}_{rows}(S),\quad O=AV.

M is zero at allowed positions and conceptually negative infinity at excluded positions. Every query needs a defined policy if all its keys are masked.

Worked example

Use q=[1,0], K=[[1,0],[0,1]], V=[[2,0],[0,2]], and d_k=2. Dot scores [1,0] become [1/sqrt(2),0]. Exponentiate and normalize to approximately [0.669762,0.330238]. The output is [2×0.669762,2×0.330238]≈[1.339523,0.660477]. If only the first key is causally allowed, the output becomes [2,0].

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sqrt,exp,isclose
q=[1,0]; keys=[[1,0],[0,1]]; values=[[2,0],[0,2]]
scores=[sum(a*b for a,b in zip(q,k))/sqrt(2) for k in keys]
weights=[exp(s-max(scores)) for s in scores]
weights=[w/sum(weights) for w in weights]
out=[sum(weights[i]*values[i][j] for i in range(2)) for j in range(2)]
assert isclose(sum(weights),1)
assert isclose(weights[0],.6697615493266569)
assert isclose(out[0],1.3395230986533138)
assert isclose(out[1],.6604769013466862)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Attention scores are not weights before normalization. Causal masking happens before softmax. Attention weights alone are not a full causal explanation of a model decision.