R7.3 · LESSON 3 OF 6

Rerankers

Fuse rankings, rerank evidence and evaluate retrieval separately from grounded answer generation.

THE IDEA

Rerankers

A reranker looks at the query and a candidate document together and scores the pair, which is usually more accurate than comparing two independently stored vectors but far more expensive. So it is applied only to the short list from earlier stages: 100 candidates at 5 ms each is 500 ms, while all 10⁶ documents would take over 80 minutes. And it can only reorder what it receives.

candidates from the first stage3

first-stage rank	id	reranker score	candidate?
1	d1	1	yes
2	d2	2	yes
3	d3	3	yes
4	d4	4	no
5	d5	9	no
6	d6	0.5	no

reranked first: d3 d5 (score 9) never reached the reranker: it cannot be recovered

ms per (query, document) pair5pairs scored in parallel1

reranking 100 candidates: 500 ms reranking 1,000,000 documents: 5000 s the joint scoring is more accurate than comparing two stored vectors, but too slow to run on everything

Change how many candidates the first stage supplies.

Predict first. Can a reranker recover a document that was never included in its candidate set?

Chapter context

RAG combines a retrieval step with a generator that uses retrieved evidence. Better generation cannot reliably repair missing or wrong evidence. Measure retrieval and answer quality as separate stages and connect citations to actual supporting passages.

TRY IT YOURSELF

Part 1. Do it hands-on

Cost a rerank by hand, then write reranking and a latency budget.

candidates from the first stage3

first-stage rank	id	reranker score	candidate?
1	d1	1	yes
2	d2	2	yes
3	d3	3	yes
4	d4	4	no
5	d5	9	no
6	d6	0.5	no

reranked first: d3 d5 (score 9) never reached the reranker: it cannot be recovered

ms per (query, document) pair5pairs scored in parallel1

reranking 100 candidates: 500 ms reranking 1,000,000 documents: 5000 s the joint scoring is more accurate than comparing two stored vectors, but too slow to run on everything

ms for 100 pairs at 5 ms seconds for 10⁶ pairs first after reranking (d1 0.2, d2 0.9, d3 0.5) can it recover a non-candidate? (yes/no)

To do: A reranker scores each (query, document) pair jointly and costs 5 ms per pair, one at a time. Reranking 100 candidates takes how many ms?
To do: Reranking all 1,000,000 documents of a corpus instead would take how many seconds?
To do: Candidates and reranker scores: d1 = 0.2, d2 = 0.9, d3 = 0.5. Which id is ranked first after reranking?
To do: Can a reranker recover a document that the first stage never put in the candidate set? Type yes or no

Part 2. Do it in code

rerank(candidates, pair_scores, k): candidates is the first-stage id list; pair_scores maps ids to the reranker's score. Return the k candidate ids with the highest score (ties: earlier in candidates). Ids not in pair_scores count as −infinity. Only candidates are considered.
recoverable(candidates, relevant): the relevant ids that are in the candidate set (sorted).
rerank_ms(n_pairs, ms_per_pair, parallel=1): ceil(n_pairs / parallel) · ms_per_pair.
max_candidates(budget_ms, ms_per_pair, parallel=1): the largest candidate count whose rerank fits the budget.
pipeline_hit(first_stage, relevant_id, n_candidates, pair_scores): True if relevant_id is ranked FIRST after reranking the top n_candidates of first_stage.

import math

def rerank(candidates, pair_scores, k):
    pass

def recoverable(candidates, relevant):
    pass

def rerank_ms(n_pairs, ms_per_pair, parallel=1):
    pass

def max_candidates(budget_ms, ms_per_pair, parallel=1):
    pass

def pipeline_hit(first_stage, relevant_id, n_candidates, pair_scores):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can a reranker recover a document never included in its candidate set?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

RRF(d)=\sum_r\frac1{k+rank_r(d)},\quad MRR=\frac1Q\sum_q\frac1{rank_q(first\ relevant)}.

An exponential-gain convention is $DCG@K=\sum_{i=1}^K(2^{rel_i}-1)/\log_2(i+1)$ and $nDCG=DCG/IDCG$ . Some tools use raw relevance gains instead: do not mix the conventions silently.

A common textbook-scaled single-term BM25 contribution is

IDF(t)\frac{tf(t,d)(k_1+1)}{tf(t,d)+k_1(1-b+b|d|/avgdl)}.

A commonly used nonnegative IDF is $\log[1+(N-df(t)+0.5)/(df(t)+0.5)]$ . Here tf is term count, df document frequency, N corpus document count and avgdl average document length. Engine-specific constant factors, field statistics and length encodings can differ, so these numbers are a teaching model rather than a promise of byte-for-byte engine scores.

Worked example

With k=60, document A has ranks 1 and 3, while B has ranks 2 and 1. A scores 1/61+1/63≈0.0322665 and B scores 1/62+1/61≈0.0325225, so B wins. For exponential gains and relevance order [1,0,2], DCG is 1+3/2=2.5, compared with ideal 3+1/log2(3).

For the stated BM25 variant, a term occurring twice in an average-length document with k1=1.2 has frequency factor 2×2.2/(2+1.2)=1.375 before multiplying by IDF. Increasing count saturates rather than increasing score linearly.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log2,isclose
A=1/61+1/63
B=1/62+1/61
assert B > A
assert isclose(A,.032266458495966696)
assert isclose(B,.03252247488101534)
dcg=lambda labels: sum((2**rel-1)/log2(i+2) for i,rel in enumerate(labels))
assert dcg([1,0,2]) == 2.5
ndcg=dcg([1,0,2])/dcg([2,1,0])
assert 0 < ndcg < 1
tf,k1,b,length,average=2,1.2,.75,100,100
factor=tf*(k1+1)/(tf+k1*(1-b+b*length/average))
assert isclose(factor,1.375)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A retrieved answer can still hallucinate. RRF's k is a fusion constant, not the result-count k in recall@k. Evaluation must state whether relevance gains are linear or exponential.