R6.2 · LESSON 2 OF 6

Fixed-size chunks

Create source-traceable chunks, calculate overlap and explain ingestion choices that preserve answer-bearing context.

THE IDEA

Fixed-size chunks

The simplest chunker cuts the text every C units. It is deterministic and easy to reason about: 1000 tokens in chunks of 200 give exactly 5 chunks, and 1050 give 6 with a short final chunk of 50. But it ignores meaning, and the unit matters: characters, words and tokens are different, so name the unit and (for tokens) pin the tokenizer.

document length (tokens)1050chunk size200

6 chunks; sizes 200, 200, 200, 200, 200, 50 stride = chunk size = 200 (adjacent, no repeated text)

Characters cut straight through words: "the quick |brown fox |jumps over| the lazy |dog". Always state the unit (characters, words or tokens under a pinned tokenizer): 200 characters and 200 tokens are different amounts of text.

Vary the document length and the chunk size.

Predict first. Are 200 characters and 200 tokens generally the same amount of text?

Chapter context

Retrieval can only find information represented in its candidate chunks. Chunking chooses the unit of retrieval; it should preserve enough context to answer a question while remaining small enough to rank and use effectively.

TRY IT YOURSELF

Part 1. Do it hands-on

Count fixed chunks by hand, then write the chunkers.

document length (tokens)1050chunk size200

6 chunks; sizes 200, 200, 200, 200, 200, 50 stride = chunk size = 200 (adjacent, no repeated text)

chunks for 1000 tokens, size 200 last chunk for 1050 tokens stride, no overlap, size 200 200 characters = 200 tokens? (yes/no)

To do: A document of 1000 tokens cut into fixed chunks of 200 tokens with no overlap: number of chunks?
To do: A document of 1050 tokens, chunk size 200, no overlap: how many tokens are in the final chunk?
To do: With no overlap, how far does each chunk start after the previous one (the stride) for chunk size 200?
To do: Are 200 characters and 200 tokens generally the same amount of text? Type yes or no

Part 2. Do it in code

chunk_units(units, size): consecutive chunks of `size` units (a list of lists); the last may be shorter. size < 1 raises ValueError.
n_chunks(n, size): how many chunks cover n units (0 for n = 0).
chunk_spans(n, size): the (start, end) pairs, end exclusive.
chunk_chars(text, size): the text cut into strings of `size` characters.
unit_counts(text): {"chars": len(text), "words": number of whitespace-separated words}.
is_lossless(text, size): True if joining chunk_chars(text, size) gives the original text back.

def chunk_units(units, size):
    pass

def n_chunks(n, size):
    pass

def chunk_spans(n, size):
    pass

def chunk_chars(text, size):
    pass

def unit_counts(text):
    pass

def is_lossless(text, size):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Are 200 characters and 200 tokens generally the same amount of text?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For document length N, maximum chunk length C, overlap O and stride $S=C-O>0$ , a coverage-stopping scheme uses

K=1+\left\lceil\frac{\max(0,N-C)}{S}\right\rceil\quad(N>0).

Stop once a chunk reaches the document end; blindly iterating start positions can create redundant trailing chunks. Define empty-document behavior separately.

Worked example

For N=1000,C=200,O=50, stride is 150 and K=1+ceil(800/150)=7. Starts are 0,150,300,450,600,750,900; the final chunk contains 100 tokens. Total stored token positions are 6×200+100=1300 because overlap duplicates context.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import ceil
N,C,O=1000,200,50
chunks=[]
start=0
while start < N:
    end=min(start+C,N)
    chunks.append((start,end))
    if end == N: break
    start += C-O
assert len(chunks) == 1+ceil(max(0,N-C)/(C-O)) == 7
assert sum(end-start for start,end in chunks) == 1300
assert chunks[-1] == (900,1000)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Overlap is not new evidence. Chunk-size advice depends on corpus and retrieval task. Retrieved passages need provenance and versioning even when their text appears self-contained.