R7.6 · LESSON 6 OF 6

Retrieval metrics

Fuse rankings, rerank evidence and evaluate retrieval separately from grounded answer generation.

THE IDEA

Retrieval metrics

Judge retrieval with graded, labelled queries. Recall@k asks how many relevant items were found; reciprocal rank uses the position of the first relevant one; nDCG weights graded relevance by rank. For relevances [1, 0, 2] with exponential gain DCG is 1 + 0 + 3/2 = 2.5 and the ideal order [2, 1, 0] gives 3 + 1/log₂3 = 3.6309, so nDCG = 0.6885; with linear gain the same ranking scores 0.7602. State your gain convention. And evaluate retrieval separately from answer generation to know which stage failed.

Click a result to change its relevance label (0 to 3).

DCG = 1/log2(2) + 0/log2(3) + 3/log2(4) = 2.5 ideal order [2,1,0]: DCG 3.6309 nDCG = 0.6885 (other gain convention: 0.7602) first relevant result at rank 1 → reciprocal rank 1

Measure retrieval and answer generation separately: if the evidence was not retrieved, fix retrieval; if it was retrieved and the answer is still wrong, fix generation.

Change the relevance labels and the gain convention.

Predict first. Can two nDCG implementations differ when one uses raw relevance and another exponential gain?

Chapter context

RAG combines a retrieval step with a generator that uses retrieved evidence. Better generation cannot reliably repair missing or wrong evidence. Measure retrieval and answer quality as separate stages and connect citations to actual supporting passages.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute DCG and nDCG by hand, then write the metrics and a retrieval-versus-generation diagnosis.

Click a result to change its relevance label (0 to 3).

DCG = 1/log2(2) + 0/log2(3) + 3/log2(4) = 2.5 ideal order [2,1,0]: DCG 3.6309 nDCG = 0.6885 (other gain convention: 0.7602) first relevant result at rank 1 → reciprocal rank 1

Measure retrieval and answer generation separately: if the evidence was not retrieved, fix retrieval; if it was retrieved and the answer is still wrong, fix generation.

DCG of [1,0,2], exponential nDCG exponential nDCG linear conventions can differ? (yes/no)

To do: Results in rank order have graded relevance [1, 0, 2]. With exponential gain (2^rel − 1) and discount log₂(rank + 1), DCG = 1/1 + 0 + 3/2. Type the value
To do: The ideal order [2, 1, 0] has DCG 3 + 1/log₂3 = 3.6309. nDCG = 2.5 / 3.6309 (4 decimals)?
To do: With linear gain (gain = rel) the same ranking has DCG 2.0 and ideal 2.6309. nDCG (4 decimals)?
To do: Can two nDCG implementations give different numbers for the same ranking when they use different gain conventions? Type yes or no

Part 2. Do it in code

recall_at_k(ranked, relevant, k): found relevant items in the first k divided by len(relevant) (0 if none are relevant).
reciprocal_rank(ranked, relevant): 1 / rank of the first relevant result (0 if none).
dcg(rels, gain="exp", k=None): Σ gain(rel) / log2(rank + 1) over the first k graded relevances in rank order; gain "exp" is 2**rel − 1, "linear" is rel. Any other gain name raises ValueError.
ndcg(rels, gain="exp", k=None): dcg divided by the dcg of the same relevances sorted best first (0 if the ideal is 0).
diagnose(retrieved_has_evidence, answer_correct): "ok" if both, "generation_failure" if the evidence was retrieved but the answer is wrong, "retrieval_failure" if the evidence was missing and the answer is wrong, "lucky" if the evidence was missing but the answer happens to be right.

import math

def recall_at_k(ranked, relevant, k):
    pass

def reciprocal_rank(ranked, relevant):
    pass

def dcg(rels, gain="exp", k=None):
    pass

def ndcg(rels, gain="exp", k=None):
    pass

def diagnose(retrieved_has_evidence, answer_correct):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can two nDCG implementations differ when one uses raw relevance and another exponential gain?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

RRF(d)=\sum_r\frac1{k+rank_r(d)},\quad MRR=\frac1Q\sum_q\frac1{rank_q(first\ relevant)}.

An exponential-gain convention is $DCG@K=\sum_{i=1}^K(2^{rel_i}-1)/\log_2(i+1)$ and $nDCG=DCG/IDCG$ . Some tools use raw relevance gains instead: do not mix the conventions silently.

A common textbook-scaled single-term BM25 contribution is

IDF(t)\frac{tf(t,d)(k_1+1)}{tf(t,d)+k_1(1-b+b|d|/avgdl)}.

A commonly used nonnegative IDF is $\log[1+(N-df(t)+0.5)/(df(t)+0.5)]$ . Here tf is term count, df document frequency, N corpus document count and avgdl average document length. Engine-specific constant factors, field statistics and length encodings can differ, so these numbers are a teaching model rather than a promise of byte-for-byte engine scores.

Worked example

With k=60, document A has ranks 1 and 3, while B has ranks 2 and 1. A scores 1/61+1/63≈0.0322665 and B scores 1/62+1/61≈0.0325225, so B wins. For exponential gains and relevance order [1,0,2], DCG is 1+3/2=2.5, compared with ideal 3+1/log2(3).

For the stated BM25 variant, a term occurring twice in an average-length document with k1=1.2 has frequency factor 2×2.2/(2+1.2)=1.375 before multiplying by IDF. Increasing count saturates rather than increasing score linearly.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log2,isclose
A=1/61+1/63
B=1/62+1/61
assert B > A
assert isclose(A,.032266458495966696)
assert isclose(B,.03252247488101534)
dcg=lambda labels: sum((2**rel-1)/log2(i+2) for i,rel in enumerate(labels))
assert dcg([1,0,2]) == 2.5
ndcg=dcg([1,0,2])/dcg([2,1,0])
assert 0 < ndcg < 1
tf,k1,b,length,average=2,1.2,.75,100,100
factor=tf*(k1+1)/(tf+k1*(1-b+b*length/average))
assert isclose(factor,1.375)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A retrieved answer can still hallucinate. RRF's k is a fusion constant, not the result-count k in recall@k. Evaluation must state whether relevance gains are linear or exponential.