R5.6 · LESSON 6 OF 6

Hybrid storage and operations

Design a retrieval record and explain filtering, indexing, partitioning, updates and consistency.

THE IDEA

Hybrid storage and operations

A production system keeps text, vectors and metadata together and consistent even when ingestion partly fails: write an ingest log, reconcile the three stores, retry what is missing. Plan backup, rebuild and model migration in advance. Changing the embedding model means re-embedding everything into a second index (extra storage while both exist) and cutting over after the backfill. Two model versions cannot be mixed safely just because their dimensions match.

chunks100,000embeddings per second500embedding failures in an ingest of 103

migration to a new model (384-d): new vectors 153,600,000 bytes while the old index keeps serving backfill time 200 s (0.06 h), then cut over last ingest of 10: 7 records complete, 3 have text but no vector → reconcile and retry

Keep text, vectors and metadata consistent under failures (write an ingestion log and reconcile). Define backup, rebuild and model-migration procedures up front. Mixing two model versions is unsafe even when dimensions match: the coordinates mean different things.

Cost a migration and an ingest failure.

Predict first. Can two embedding-model versions be mixed safely solely because their dimensions match?

Chapter context

A vector index solves similarity lookup; a retrieval application also needs durable records, metadata, permissions and updates. Database behavior determines which vectors are eligible and which version of a document an answer uses.

TRY IT YOURSELF

Part 1. Do it hands-on

Cost a migration by hand, then write the consistency check and migration plan.

chunks100,000embeddings per second500embedding failures in an ingest of 103

extra bytes, 100k × 384 float32 backfill seconds at 500/s records with text but no vector mix model versions with equal dims? (yes/no)

To do: Migrating 100,000 chunks to a new 384-dimensional model while the old index keeps serving: extra bytes for the new float32 vectors?
To do: Backfilling at 500 chunks per second takes how many seconds?
To do: An ingestion run stored text and metadata for 10 chunks but embedding failed for 3. How many records have text but no vector?
To do: Two embedding-model versions give vectors of the same dimension. Can they be mixed in one index safely? Type yes or no

Part 2. Do it in code

reconcile(text_ids, vector_ids, meta_ids): compare three collections of ids; return a dict with sorted lists: missing_vectors (in text but not vectors), orphan_vectors (in vectors but not text), missing_meta (in text but not meta).
safe_to_mix(model_a, dim_a, model_b, dim_b): True only if the model names AND the dimensions are equal.
migration_plan(old_model, old_dim, new_model, new_dim): a dict {"needs_reembed": bool, "dual_index": bool, "cutover": str}: needs_reembed is True if the model or dimension changes; dual_index equals needs_reembed; cutover is "after_backfill" when re-embedding is needed, otherwise "immediate".
reembed_seconds(n_chunks, per_second): time to re-embed (n_chunks / per_second).
extra_storage_bytes(n_chunks, new_dim, float_bytes=4): bytes for the new vectors while the old index still serves.

def reconcile(text_ids, vector_ids, meta_ids):
    pass

def safe_to_mix(model_a, dim_a, model_b, dim_b):
    pass

def migration_plan(old_model, old_dim, new_model, new_dim):
    pass

def reembed_seconds(n_chunks, per_second):
    pass

def extra_storage_bytes(n_chunks, new_dim, float_bytes=4):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can two embedding-model versions be mixed safely solely because their dimensions match?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

Raw vector storage is $M=Ndb$ for N vectors of dimension d and b bytes per coordinate. If a postfilter independently accepts fraction f of an initial k candidates, the expected survivor count is kf; this is an expectation, not a guarantee.

Worked example

Ten million vectors of dimension 768 in float32 require 10,000,000×768×4=30,720,000,000 bytes, or 30.72 decimal GB, before IDs, text, indexes and replicas. If a postfilter accepts 20% of ten candidates under a simplifying independence assumption, the expected survivor count is only two.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
bytes_used=10_000_000*768*4
assert bytes_used == 30_720_000_000
assert isclose(bytes_used/1_000_000_000,30.72)
assert 10*.2 == 2

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Filter selectivity and ANN candidate recall interact. Replication adds storage. Matching vector dimension does not establish matching embedding space.