Retention and deletion
Choose representations and storage boundaries from access patterns, ownership and retention requirements.
THE IDEA
Retention and deletion
Decide how long data, derived views and backups are kept, and what "deleted" must mean. A simple storage estimate is writes per day × bytes × days × copies: 2,000,000 records of 500 bytes is 1 GB a day, so 30 days in three full copies is 90 GB, and 90 days would be 270 GB, before indexes and backups. Deleting a source row alone leaves copies in caches and indexes; and a replica is not a backup, because it copies mistakes and deletions too.
State how long data, derived views and backups are kept. A replica copies deletes too, so it cannot stand in for an independent, recoverable backup.
Predict first. Is a database replica a substitute for an independently recoverable backup?
Chapter context
A data model describes the facts a system owns and the relationships it must preserve. Storage choices should follow how those facts are queried, updated and retained, rather than a blanket preference for one database category.
TRY IT YOURSELF
Part 1. Do it hands-on
Size retained data by hand, then write expiry, a full purge and a leak check.
State how long data, derived views and backups are kept. A replica copies deletes too, so it cannot stand in for an independent, recoverable backup.
- To do: 2,000,000 records a day at 500 bytes each: decimal GB of raw payload per day?
- To do: Kept 30 days in three full copies: decimal GB (before indexes and backups)?
- To do: If retention is raised to 90 days: decimal GB?
- To do: Is a database replica a substitute for an independently recoverable backup? Type yes or no
Part 2. Do it in code
- retained_bytes(writes_per_day, payload_bytes, days, copies): w · s · d · r.
- to_gb(n_bytes): decimal gigabytes.
- expired(records, now_day, retention_days): records are {"id", "day"} (the day created); return the sorted ids older than the retention: now_day − day > retention_days.
- purge(source, derived_views, ids): remove the ids from the source dict AND from every derived view (a list of dicts such as caches or indexes). Return the number of copies removed in total.
- leaked(source, derived_views, deleted_ids): the sorted ids that were deleted from the source but still appear in some derived view.
- restorable(has_backup, backup_age_days, max_age_days): True only if there is an independent backup no older than max_age_days (a replica is not a backup).
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Is a database replica a substitute for an independently recoverable backup?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
A simple retained-payload estimate is
where w is writes/day, s payload bytes/write, d retained days and r full-copy count. Add indexes, metadata, compaction headroom and backups separately; replication schemes may store more than exactly r payload copies.
Worked example
Two million records per day at 500 bytes each produce one decimal GB/day of raw payload. Thirty days retained across three full copies require 2,000,000×500×30×3=90,000,000,000 bytes, or 90 decimal GB before other storage. Compressibility and index size are separate measured factors.
Reference code
The following example uses the Python standard library and does not call a remote service.
daily,size,days,copies=2_000_000,500,30,3
assert daily*size == 1_000_000_000
retained=daily*size*days*copies
assert retained == 90_000_000_000
assert retained/1e9 == 90
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
NoSQL is not a single consistency or query model. Denormalization creates a maintenance responsibility. Payload estimates must not silently stand in for provisioned disk requirements.