R6.1 · LESSON 1 OF 6

Parsing and structure

Create source-traceable chunks, calculate overlap and explain ingestion choices that preserve answer-bearing context.

THE IDEA

Parsing and structure

Before chunking, the source (HTML, PDF, Markdown, a spreadsheet) must become text. How you do it decides what survives. A table flattened to "Plan Price Seats Pro 20 5 Team 50 25" no longer says which number is a price. Keep headings, list structure and table rows as self-contained units, and record provenance (file, section, line) so a parser error can be traced to its cause.

Plan	Price	Seats
Pro	20	5
Team	50	25

row for the question:

what the retriever sees: "Plan Price Seats Pro 20 5 Team 50 25" which of 50 / 25 / 5 is the price of Team? the column of each number is gone

Splitting at headings gives one section per heading (3 here). Record provenance (file, section, line number) so parser mistakes can be traced.

Switch between flattened and structure-preserving extraction.

Predict first. Can flattening a table destroy relationships needed for an answer?

Chapter context

Retrieval can only find information represented in its candidate chunks. Chunking chooses the unit of retrieval; it should preserve enough context to answer a question while remaining small enough to rank and use effectively.

TRY IT YOURSELF

Part 1. Do it hands-on

Read a table by hand, then write a parser that keeps its structure.

Plan	Price	Seats
Pro	20	5
Team	50	25

row for the question:

what the retriever sees: "Plan Price Seats Pro 20 5 Team 50 25" which of 50 / 25 / 5 is the price of Team? the column of each number is gone

Splitting at headings gives one section per heading (3 here). Record provenance (file, section, line number) so parser mistakes can be traced.

Price of Team Team row sentence sections from 3 headings can flattening destroy relationships? (yes/no)

To do: A pricing table has columns Plan, Price, Seats and rows (Pro, 20, 5) and (Team, 50, 25). A parser flattens it to: Plan Price Seats Pro 20 5 Team 50 25. From the table itself, what is the Price of Team?
To do: Write the Team row as a self-contained line of the form Plan: Team; Price: 50; Seats: 25 (type exactly that shape)
To do: A document has 3 headings, so splitting at headings gives how many sections?
To do: Can flattening a table destroy relationships needed for an answer? Type yes or no

Part 2. Do it in code

parse_table(md): a markdown table (lines like "| a | b |", the second line a separator of dashes) into a list of row dicts keyed by the header cells. Cell text is stripped. Fewer than 2 lines raises ValueError.
flatten_table(md): all cell texts of all rows (header first, separator skipped) joined by single spaces: this is what a careless parser produces.
row_sentence(row): "Header: value; Header: value" in header order.
split_sections(md): split text at lines starting with '#'; return a list of {"heading": heading text without the #s, "line": 1-based line number of the heading, "text": the body lines joined by "\n"}. Text before the first heading is ignored.
lookup(rows, key_col, key, value_col): the value_col of the row whose key_col equals key, or None.

def parse_table(md):
    pass

def flatten_table(md):
    pass

def row_sentence(row):
    pass

def split_sections(md):
    pass

def lookup(rows, key_col, key, value_col):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can flattening a table destroy relationships needed for an answer?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For document length N, maximum chunk length C, overlap O and stride $S=C-O>0$ , a coverage-stopping scheme uses

K=1+\left\lceil\frac{\max(0,N-C)}{S}\right\rceil\quad(N>0).

Stop once a chunk reaches the document end; blindly iterating start positions can create redundant trailing chunks. Define empty-document behavior separately.

Worked example

For N=1000,C=200,O=50, stride is 150 and K=1+ceil(800/150)=7. Starts are 0,150,300,450,600,750,900; the final chunk contains 100 tokens. Total stored token positions are 6×200+100=1300 because overlap duplicates context.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import ceil
N,C,O=1000,200,50
chunks=[]
start=0
while start < N:
    end=min(start+C,N)
    chunks.append((start,end))
    if end == N: break
    start += C-O
assert len(chunks) == 1+ceil(max(0,N-C)/(C-O)) == 7
assert sum(end-start for start,end in chunks) == 1300
assert chunks[-1] == (900,1000)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Overlap is not new evidence. Chunk-size advice depends on corpus and retrieval task. Retrieved passages need provenance and versioning even when their text appears self-contained.