R6.5 · LESSON 5 OF 6

Metadata and contextual headers

Create source-traceable chunks, calculate overlap and explain ingestion choices that preserve answer-bearing context.

THE IDEA

Metadata and contextual headers

A passage cut from its document loses its context: "Pay within 30 days." of what, in which version? Store metadata with each chunk (source, version, section, chunk index) for filtering and traceability, and consider prepending a short contextual header before embedding. A 12-token header on a 200-token chunk is a 6% overhead.

document section

version3chunk tokens200

bare chunk: "Pay within 30 days." (which document? which version?) with the header: "Doc: Handbook (v3) | Section: Refunds\nPay within 30 days." metadata: { doc: "Handbook", version: 3, section: "Refunds", chunk_index: 4, source: "docs/handbook.md" } header of 12 tokens on a 200-token chunk = 0.06 extra (212 tokens embedded)

The header restores context lost when a passage is cut from its document. It must only restate what the source says, never add facts. Metadata (version, section, source) also lets you filter out old or ineligible content.

Edit the document, section and version.

Predict first. Should a header invent facts that are missing from the source?

Chapter context

Retrieval can only find information represented in its candidate chunks. Chunking chooses the unit of retrieval; it should preserve enough context to answer a question while remaining small enough to rank and use effectively.

TRY IT YOURSELF

Part 1. Do it hands-on

Cost a header by hand, then write the header and metadata helpers.

document section

version3chunk tokens200

tokens embedded per chunk extra tokens for 7 chunks overhead 12/200 may headers invent facts? (yes/no)

To do: A 200-token chunk gets a 12-token contextual header (document, version, section) before it is embedded. Tokens embedded per chunk?
To do: For the 7 chunks of a document, extra tokens embedded because of the headers?
To do: Header overhead as a fraction of the chunk (12 / 200)?
To do: Should a contextual header be allowed to add facts that are not in the source? Type yes or no

Part 2. Do it in code

make_header(doc, version, section=None): "Doc: <doc> (v<version>)" followed by " | Section: <section>" when a non-empty section is given.
with_header(chunk_text, header): header + "\n" + chunk_text.
make_metadata(doc, version, section, chunk_index, source): a dict with those keys: doc, version, section, chunk_index, source.
eligible(meta, min_version, section=None): True if meta['version'] >= min_version and (section is None or meta['section'] == section).
header_overhead(header_tokens, chunk_tokens): header_tokens / chunk_tokens.
facts_supported(header_fields, source_fields): True only if every key in header_fields exists in source_fields with the same value (a header must not invent facts).

def make_header(doc, version, section=None):
    pass

def with_header(chunk_text, header):
    pass

def make_metadata(doc, version, section, chunk_index, source):
    pass

def eligible(meta, min_version, section=None):
    pass

def header_overhead(header_tokens, chunk_tokens):
    pass

def facts_supported(header_fields, source_fields):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Should a header invent facts missing from the source?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For document length N, maximum chunk length C, overlap O and stride $S=C-O>0$ , a coverage-stopping scheme uses

K=1+\left\lceil\frac{\max(0,N-C)}{S}\right\rceil\quad(N>0).

Stop once a chunk reaches the document end; blindly iterating start positions can create redundant trailing chunks. Define empty-document behavior separately.

Worked example

For N=1000,C=200,O=50, stride is 150 and K=1+ceil(800/150)=7. Starts are 0,150,300,450,600,750,900; the final chunk contains 100 tokens. Total stored token positions are 6×200+100=1300 because overlap duplicates context.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import ceil
N,C,O=1000,200,50
chunks=[]
start=0
while start < N:
    end=min(start+C,N)
    chunks.append((start,end))
    if end == N: break
    start += C-O
assert len(chunks) == 1+ceil(max(0,N-C)/(C-O)) == 7
assert sum(end-start for start,end in chunks) == 1300
assert chunks[-1] == (900,1000)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Overlap is not new evidence. Chunk-size advice depends on corpus and retrieval task. Retrieved passages need provenance and versioning even when their text appears self-contained.