R5.1 · LESSON 1 OF 6

Records and identity

Design a retrieval record and explain filtering, indexing, partitioning, updates and consistency.

THE IDEA

Records and identity

A vector database stores more than numbers. Each record should carry a stable identifier (document and chunk), the identity of the embedding model that produced the vector, the text or a pointer to it, and useful metadata for filtering. A vector with no traceable source cannot be audited, updated or deleted reliably.

document id embedding model

chunk number3

record { id: "handbook#3" model: "embed-v1" dim: 384 vector: [0.12, −0.03, …] (384 floats = 1,536 bytes) text: "…the passage…" metadata: { lang: "en", updated: "2025-01-10" } } this vector can be traced to its passage, updated or deleted by id

12,500 documents with 8 chunks each are 100,000 records. At 384 float32 dimensions the vectors alone are 153,600,000 bytes. Record the embedding-model identity so you know which vectors can be compared.

Toggle the source id and metadata and see what the record can and cannot do.

Predict first. What links a retrieved vector back to a passage?

Chapter context

A vector index solves similarity lookup; a retrieval application also needs durable records, metadata, permissions and updates. Database behavior determines which vectors are eligible and which version of a document an answer uses.

TRY IT YOURSELF

Part 1. Do it hands-on

Size and name records by hand, then write id helpers and a record validator.

document id embedding model

chunk number3

id of chunk 3 of "handbook" records (12,500 × 8) vector bytes (384-d) traceable without an id? (yes/no)

To do: A stable chunk identifier combines the document id and the chunk number as doc#chunk. Type the id of chunk 3 of the document "handbook"
To do: 12,500 documents, 8 chunks each: how many vector records?
To do: With a 384-dimensional float32 vector per record, bytes for the vectors alone?
To do: A vector stored with no source id can be traced back to its passage. Type yes or no

Part 2. Do it in code

make_id(doc_id, chunk_index): "doc#chunk". ValueError for an empty doc_id, a doc_id containing '#', or a negative chunk index.
parse_id(chunk_id): (doc_id, chunk_index as int); ValueError if the id is not of that form.
content_id(text): a stable 8-character id from the text itself: the first 8 hex characters of sha1(text.encode()). Same text gives the same id, any change gives a different one.
validate_record(rec, model, dim): a list of problems in this order: "missing:<key>" for each key of id, doc_id, model, vector, text that is absent; then "model_mismatch" if rec['model'] != model; then "dim_mismatch" if len(rec['vector']) != dim. (Skip a check when its key is missing.) A good record gives [].
trace(store, chunk_id): store maps ids to records; return the record's text, or None if the id is unknown.

import hashlib

def make_id(doc_id, chunk_index):
    pass

def parse_id(chunk_id):
    pass

def content_id(text):
    pass

def validate_record(rec, model, dim):
    pass

def trace(store, chunk_id):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

What links a retrieved vector back to a passage?

AThe similarity score returned with the hit.BA stable source and chunk identifier stored with the vector, since rank positions change from query to query.CThe position of the vector in the nearest-neighbour list.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

Raw vector storage is $M=Ndb$ for N vectors of dimension d and b bytes per coordinate. If a postfilter independently accepts fraction f of an initial k candidates, the expected survivor count is kf; this is an expectation, not a guarantee.

Worked example

Ten million vectors of dimension 768 in float32 require 10,000,000×768×4=30,720,000,000 bytes, or 30.72 decimal GB, before IDs, text, indexes and replicas. If a postfilter accepts 20% of ten candidates under a simplifying independence assumption, the expected survivor count is only two.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
bytes_used=10_000_000*768*4
assert bytes_used == 30_720_000_000
assert isclose(bytes_used/1_000_000_000,30.72)
assert 10*.2 == 2

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Filter selectivity and ANN candidate recall interact. Replication adds storage. Matching vector dimension does not establish matching embedding space.