R6.6 · LESSON 6 OF 6

Incremental ingestion

Create source-traceable chunks, calculate overlap and explain ingestion choices that preserve answer-bearing context.

THE IDEA

Incremental ingestion

Re-embedding the whole corpus on every change is wasteful. Hash each document, compare with the hashes already indexed, and process only the difference: added, changed and removed documents. With 1000 documents and a sync of 30 new, 50 changed and 20 removed, 80 documents (640 chunks at 8 each) are embedded and 930 are untouched. Afterwards verify that what is indexed matches the source.

new documents30changed documents50removed documents20chunks per document8

index before: 1000 documents documents to embed: 30 new + 50 changed = 80 (640 chunks) unchanged and untouched: 930 removed: 20 index after: 1010 only the changed content is re-embedded; everything else keeps its vectors

Detect change with a content hash, version each chunk, and verify afterwards: compare the indexed hash with the current source hash. A successful embedding call only proves a vector was produced. Test the pipeline with questions whose evidence position is known.

Change the size of the sync.

Predict first. Does a successful embedding call prove the correct source version was indexed?

Chapter context

Retrieval can only find information represented in its candidate chunks. Chunking chooses the unit of retrieval; it should preserve enough context to answer a question while remaining small enough to rank and use effectively.

TRY IT YOURSELF

Part 1. Do it hands-on

Plan a sync by hand, then write the diff, apply and verify steps.

new documents30changed documents50removed documents20chunks per document8

documents to embed (30 new, 50 changed) unchanged of the 1000 chunks to embed (8 per document) embed call success proves the right version? (yes/no)

To do: An index holds 1000 documents. On the next sync 30 are new, 50 changed and 20 were removed. How many documents must be (re-)embedded?
To do: How many of the 1000 existing documents are unchanged (neither changed nor removed)?
To do: At 8 chunks per document, chunks to embed on this sync?
To do: Does a successful embedding call prove the correct source version is now indexed? Type yes or no

Part 2. Do it in code

content_hash(text): the first 12 hex characters of sha1(text.encode()).
ingest_plan(old, new): old and new map document ids to content hashes. Return {"added", "changed", "removed", "unchanged"}, each a sorted list of ids.
apply_plan(old, plan, new): a new index (dict) after the plan: removed ids are dropped; added and changed ids take their hash from new; unchanged keep theirs.
stale_documents(index, source_hashes): ids whose indexed hash differs from the current source hash, or that are missing from the index, sorted.
chunks_to_embed(plan, chunks_per_doc): (added + changed) · chunks_per_doc.

import hashlib

def content_hash(text):
    pass

def ingest_plan(old, new):
    pass

def apply_plan(old, plan, new):
    pass

def stale_documents(index, source_hashes):
    pass

def chunks_to_embed(plan, chunks_per_doc):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does a successful embed call prove the correct source version was indexed?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For document length N, maximum chunk length C, overlap O and stride $S=C-O>0$ , a coverage-stopping scheme uses

K=1+\left\lceil\frac{\max(0,N-C)}{S}\right\rceil\quad(N>0).

Stop once a chunk reaches the document end; blindly iterating start positions can create redundant trailing chunks. Define empty-document behavior separately.

Worked example

For N=1000,C=200,O=50, stride is 150 and K=1+ceil(800/150)=7. Starts are 0,150,300,450,600,750,900; the final chunk contains 100 tokens. Total stored token positions are 6×200+100=1300 because overlap duplicates context.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import ceil
N,C,O=1000,200,50
chunks=[]
start=0
while start < N:
    end=min(start+C,N)
    chunks.append((start,end))
    if end == N: break
    start += C-O
assert len(chunks) == 1+ceil(max(0,N-C)/(C-O)) == 7
assert sum(end-start for start,end in chunks) == 1300
assert chunks[-1] == (900,1000)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Overlap is not new evidence. Chunk-size advice depends on corpus and retrieval task. Retrieved passages need provenance and versioning even when their text appears self-contained.