R6.3 · LESSON 3 OF 6

Overlap

Create source-traceable chunks, calculate overlap and explain ingestion choices that preserve answer-bearing context.

THE IDEA

Overlap

With overlap O, each chunk repeats the last O units of the previous one, so a sentence cut at a boundary still appears whole in one chunk. The start of each chunk moves by the stride = C − O. For N = 1000, C = 200, O = 50: stride 150, K = 1 + ceil(800/150) = 7 chunks starting at 0, 150, …, 900, the last with 100 tokens. Stored positions are 6 × 200 + 100 = 1300, so overlap costs 30% more storage here and can return duplicate evidence.

document tokens N1000chunk size C200overlap O50

stride = 200 − 50 = 150 chunks K = 7 starts 0, 150, 300, 450, 600, 750, 900 last chunk 100 tokens stored positions 1300 (1.3× the document: overlap duplicates context)

Chunk 1 of 7

chunk 1: tokens 0 to 200

Change N, C and O and step through the chunks.

Predict first. With chunk size 200 and overlap 50, what is the stride?

Chapter context

Retrieval can only find information represented in its candidate chunks. Chunking chooses the unit of retrieval; it should preserve enough context to answer a question while remaining small enough to rank and use effectively.

TRY IT YOURSELF

Part 1. Do it hands-on

Work out the worked example by hand, then write overlapping chunking.

document tokens N1000chunk size C200overlap O50

stride = 200 − 50 = 150 chunks K = 7 starts 0, 150, 300, 450, 600, 750, 900 last chunk 100 tokens stored positions 1300 (1.3× the document: overlap duplicates context)

Chunk 1 of 7

chunk 1: tokens 0 to 200

stride (200, overlap 50) chunks K for N = 1000 tokens in the last chunk total stored positions

To do: Chunk size 200 with overlap 50: stride?
To do: For a 1000-token document: chunks K = 1 + ceil((1000 − 200) / 150) = ?
To do: The chunks start at 0, 150, 300, 450, 600, 750, 900. Tokens in the last chunk?
To do: Total stored token positions (6 full chunks of 200 plus the last)?

Part 2. Do it in code

stride(size, overlap): size − overlap. overlap < 0 or overlap >= size raises ValueError.
chunk_starts(n, size, overlap): start offsets: begin at 0, step by the stride, stop after the first chunk whose end reaches n (empty for n = 0).
n_chunks(n, size, overlap): len(chunk_starts(...)).
chunk_lengths(n, size, overlap): the length of each chunk (the last may be shorter).
stored_positions(n, size, overlap): the sum of chunk lengths (positions kept, counting duplicates).
duplication_factor(n, size, overlap): stored positions divided by n.
chunk_with_overlap(units, size, overlap): the chunks themselves (lists).

def stride(size, overlap):
    pass

def chunk_starts(n, size, overlap):
    pass

def n_chunks(n, size, overlap):
    pass

def chunk_lengths(n, size, overlap):
    pass

def stored_positions(n, size, overlap):
    pass

def duplication_factor(n, size, overlap):
    pass

def chunk_with_overlap(units, size, overlap):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

With size 200 and overlap 50, what is the stride?

A250: the chunk size plus the overlap, 200 + 50.B50: the stride is the overlap itself.C150: the stride is the chunk size minus the overlap, 200 − 50.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For document length N, maximum chunk length C, overlap O and stride $S=C-O>0$ , a coverage-stopping scheme uses

K=1+\left\lceil\frac{\max(0,N-C)}{S}\right\rceil\quad(N>0).

Stop once a chunk reaches the document end; blindly iterating start positions can create redundant trailing chunks. Define empty-document behavior separately.

Worked example

For N=1000,C=200,O=50, stride is 150 and K=1+ceil(800/150)=7. Starts are 0,150,300,450,600,750,900; the final chunk contains 100 tokens. Total stored token positions are 6×200+100=1300 because overlap duplicates context.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import ceil
N,C,O=1000,200,50
chunks=[]
start=0
while start < N:
    end=min(start+C,N)
    chunks.append((start,end))
    if end == N: break
    start += C-O
assert len(chunks) == 1+ceil(max(0,N-C)/(C-O)) == 7
assert sum(end-start for start,end in chunks) == 1300
assert chunks[-1] == (900,1000)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Overlap is not new evidence. Chunk-size advice depends on corpus and retrieval task. Retrieved passages need provenance and versioning even when their text appears self-contained.