R7.5 · LESSON 5 OF 6

Grounding and citations

Fuse rankings, rerank evidence and evaluate retrieval separately from grounded answer generation.

THE IDEA

Grounding and citations

A grounded answer makes claims that the retrieved passages actually support, and points each claim at the passage (ideally a span) that does. A citation marker or a source title is not evidence by itself: if the linked passage does not contain the claim, the answer is not grounded. A simple check compares the claim's key terms with the cited passage; real systems use stronger tests, but the idea is the same.

Claim: "Refunds are available for 30 days." with key terms refund, 30, days.

cited passage:

passage A: "Customers get a refund within 30 days." passage B: "Shipping takes 30 days."

support threshold0.8

support = 1 → supported by the cited passage

Ask for claims supported by retrieved passages and connect each citation to a source span. A source title alone, or a marker pointing at a passage that does not contain the claim, does not make an answer grounded.

Cite passage A, passage B or nothing.

Predict first. Does attaching a source title make an unsupported answer grounded?

Chapter context

RAG combines a retrieval step with a generator that uses retrieved evidence. Better generation cannot reliably repair missing or wrong evidence. Measure retrieval and answer quality as separate stages and connect citations to actual supporting passages.

TRY IT YOURSELF

Part 1. Do it hands-on

Check support by hand, then write a citation checker.

Claim: "Refunds are available for 30 days." with key terms refund, 30, days.

cited passage:

passage A: "Customers get a refund within 30 days." passage B: "Shipping takes 30 days."

support threshold0.8

support = 1 → supported by the cited passage

support in passage A support in passage B B supports it at threshold 0.8? (yes/no) a source title makes it grounded? (yes/no)

To do: Claim key terms: refund, 30, days. Cited passage A: "Customers get a refund within 30 days." Fraction of the claim's terms found in A?
To do: Cited passage B: "Shipping takes 30 days." Fraction found in B (3 decimals)?
To do: With a support threshold of 0.8, is the claim supported by passage B? Type yes or no
To do: Does attaching a source title to an unsupported answer make it grounded? Type yes or no

Part 2. Do it in code

support_fraction(terms, passage): the share of the claim's key terms that appear as whole words (case-insensitive, split on non-alphanumerics) in the passage.
is_supported(terms, passage, threshold=0.8): support_fraction >= threshold.
check_answer(claims, passages, threshold=0.8): claims is a list of {"id", "terms", "cites"} (cites = passage ids); passages maps id → text. Return {"unsupported": [...], "uncited": [...]}: a claim with no citations is uncited; a cited claim is unsupported if NONE of its cited passages supports it (a missing passage id never supports). Lists in claim order.
find_span(passage, quote): (start, end) of the first occurrence of the quote (end exclusive), or None, so a citation can point at a source span.
grounded_rate(claims, passages, threshold=0.8): the fraction of claims that are cited AND supported (0 if no claims).

import re

def support_fraction(terms, passage):
    pass

def is_supported(terms, passage, threshold=0.8):
    pass

def check_answer(claims, passages, threshold=0.8):
    pass

def find_span(passage, quote):
    pass

def grounded_rate(claims, passages, threshold=0.8):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does attaching a source title make an unsupported answer grounded?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

RRF(d)=\sum_r\frac1{k+rank_r(d)},\quad MRR=\frac1Q\sum_q\frac1{rank_q(first\ relevant)}.

An exponential-gain convention is $DCG@K=\sum_{i=1}^K(2^{rel_i}-1)/\log_2(i+1)$ and $nDCG=DCG/IDCG$ . Some tools use raw relevance gains instead: do not mix the conventions silently.

A common textbook-scaled single-term BM25 contribution is

IDF(t)\frac{tf(t,d)(k_1+1)}{tf(t,d)+k_1(1-b+b|d|/avgdl)}.

A commonly used nonnegative IDF is $\log[1+(N-df(t)+0.5)/(df(t)+0.5)]$ . Here tf is term count, df document frequency, N corpus document count and avgdl average document length. Engine-specific constant factors, field statistics and length encodings can differ, so these numbers are a teaching model rather than a promise of byte-for-byte engine scores.

Worked example

With k=60, document A has ranks 1 and 3, while B has ranks 2 and 1. A scores 1/61+1/63≈0.0322665 and B scores 1/62+1/61≈0.0325225, so B wins. For exponential gains and relevance order [1,0,2], DCG is 1+3/2=2.5, compared with ideal 3+1/log2(3).

For the stated BM25 variant, a term occurring twice in an average-length document with k1=1.2 has frequency factor 2×2.2/(2+1.2)=1.375 before multiplying by IDF. Increasing count saturates rather than increasing score linearly.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log2,isclose
A=1/61+1/63
B=1/62+1/61
assert B > A
assert isclose(A,.032266458495966696)
assert isclose(B,.03252247488101534)
dcg=lambda labels: sum((2**rel-1)/log2(i+2) for i,rel in enumerate(labels))
assert dcg([1,0,2]) == 2.5
ndcg=dcg([1,0,2])/dcg([2,1,0])
assert 0 < ndcg < 1
tf,k1,b,length,average=2,1.2,.75,100,100
factor=tf*(k1+1)/(tf+k1*(1-b+b*length/average))
assert isclose(factor,1.375)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A retrieved answer can still hallucinate. RRF's k is a fusion constant, not the result-count k in recall@k. Evaluation must state whether relevance gains are linear or exponential.