R1.6 · LESSON 6 OF 6

Retrieval evaluation

Rank documents by dot product, cosine and L2 distance and explain the role of normalization and model choice.

THE IDEA

Evaluating retrieval

A pretty 2-D map of clusters or a high similarity score proves little. Evaluate with judged data: queries paired with the documents that really answer them. Common measures: recall@k (the share of relevant documents found in the top k) and mean reciprocal rank (average of 1/rank of the first relevant result). Include difficult near-miss documents (hard negatives) and the kinds of queries real users ask.

k (results considered)3

query	ranked results	relevant (judged)	recall@3	reciprocal rank
q1	d9d1d4	d1	1	0.5
q2	d2d7d5	d2	1	1
q3	d3d8d6	d5	0	0

recall@3 averaged: 0.667 mean reciprocal rank: 0.5 A high similarity score is not proof of relevance: evaluate with judged queries, hard negatives (near-miss documents) and the questions real users ask.

Three judged queries. Change k.

Predict first. Is a high similarity score proof that a document answers the query?

Chapter context

Semantic search represents queries and documents as vectors in a compatible space. Good retrieval depends on the encoder, its training objective and the candidate data; choosing a distance formula alone does not create semantic relevance.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute recall and MRR for three judged queries by hand, then write the evaluation code.

k (results considered)3

query	ranked results	relevant (judged)	recall@3	reciprocal rank
q1	d9d1d4	d1	1	0.5
q2	d2d7d5	d2	1	1
q3	d3d8d6	d5	0	0

average recall@3 mean reciprocal rank fraction with the relevant document first

To do: Three test queries with judged relevant documents. q1: results d9, d1, d4 (relevant d1). q2: d2, d7, d5 (relevant d2). q3: d3, d8, d6 (relevant d5). Recall@3 averaged over queries (3 decimals)?
To do: Mean reciprocal rank (1/rank of the first relevant result; 0 if none): q1 → 1/2, q2 → 1, q3 → 0. MRR?
To do: Fraction of queries whose relevant document is ranked FIRST?

Part 2. Do it in code

recall_at_k(ranked, relevant, k): the fraction of the relevant documents that appear in the first k results (0 if there are none).
reciprocal_rank(ranked, relevant): 1 / rank (1-based) of the first relevant result, or 0 if none.
mean(values): the average (0 for an empty list).
evaluate(runs, k): runs is a list of dicts with ranked and relevant. Return a dict with recall_at_k, mrr and top1 (the fraction of queries whose first result is relevant), each averaged over the runs.
with_hard_negative(ranked, negative, position): a copy of the ranking with the 'hard negative' document inserted at `position` (0-based); it pushes everything below it down.

def recall_at_k(ranked, relevant, k):
    pass

def reciprocal_rank(ranked, relevant):
    pass

def mean(values):
    pass

def evaluate(runs, k):
    pass

def with_hard_negative(ranked, negative, position):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Is a high similarity score proof that a document answers the query?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

\cos(q,x)=\frac{q^Tx}{\|q\|\|x\|},\quad \|q-x\|^2=\|q\|^2+\|x\|^2-2q^Tx.

For unit-normalized vectors, squared L2 is $2-2q^Tx$ , so their rankings agree under the appropriate direction. Zero vectors require a defined handling policy.

Worked example

Let q=[1,0], A=[1,0] and B=[10,10]. Raw dot products are 1 and 10, so B ranks first by dot product. Cosines are 1 and 1/sqrt(2), so A ranks first by cosine. This is a deliberate example of magnitude changing the retrieval objective.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sqrt,isclose
q,a,b=[1,0],[1,0],[10,10]
dot=lambda u,v: sum(x*y for x,y in zip(u,v))
norm=lambda u: sqrt(dot(u,u))
assert dot(q,b) > dot(q,a)
assert dot(q,a)/(norm(q)*norm(a)) > dot(q,b)/(norm(q)*norm(b))
assert isclose(dot(q,b)/(norm(q)*norm(b)),1/sqrt(2))
assert 128*4 == 512

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Do not mix embedding spaces or metric directions. Normalize only when it matches the model and search objective. Similarity is not factual verification.