SD10.5 · LESSON 5 OF 6

Recovery and backups

Define service objectives and use telemetry, failure handling and recovery procedures to protect user outcomes.

THE IDEA

Recovery and backups

Two numbers define recovery. The recovery point objective (RPO) is how much data you can afford to lose: with 6-hourly backups and nothing else, up to 6 hours; with logs shipped every 5 minutes, about 5 minutes. The recovery time objective (RTO) is how long restoration may take: restoring 2 TB at 500 MB/s needs at least 4,000 seconds. A replica is not a backup because it copies deletions and corruption too. And a green "backup completed" light proves nothing until a restore has been tested.

backup interval (hours)6data size (TB)2restore speed (MB/s)500

worst-case data loss (the recovery point you actually achieve): 6 hours restore time lower bound: 4000 s = 1.11 h (the recovery time you actually achieve) compare both with the objectives you promised (RPO for loss, RTO for restore time) a replica copies a bad DELETE within seconds, so only a backup or a log from before it can undo it

A finished backup job tells you a job ended, not that you can restore from it. Test restores, time them, and rehearse the failure procedure.

Change the backup interval, size and speed.

Predict first. Does a replica protect against every accidental logical deletion?

Chapter context

Reliability is measured against what users need, not whether every component reports healthy. Observability connects symptoms to evidence, while resilience and recovery determine what happens when dependencies or infrastructure fail.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute loss and restore time by hand, then write the recovery helpers.

backup interval (hours)6data size (TB)2restore speed (MB/s)500

A finished backup job tells you a job ended, not that you can restore from it. Test restores, time them, and rehearse the failure procedure.

worst-case loss with 6-hourly backups (h) with 5-minute logs (min) seconds to restore 2 TB at 500 MB/s a replica protects against logical deletion? (yes/no)

To do: Full backups run every 6 hours and nothing else. In the worst case, how many hours of data could be lost (the recovery point objective achieved)?
To do: With transaction logs shipped every 5 minutes (point-in-time recovery), worst-case data loss in minutes?
To do: Restoring 2 TB (2 × 10¹² bytes) at 500 MB/s (5 × 10⁸ B/s) takes how many seconds? (a lower bound for the recovery time objective)
To do: Does a replica protect against every accidental logical deletion? Type yes or no

Part 2. Do it in code

worst_case_loss_hours(backup_interval_hours, log_interval_minutes=None): with only periodic full backups the worst-case loss is the backup interval (hours); with log shipping it is the log interval (minutes ÷ 60 hours).
restore_seconds(size_bytes, bytes_per_second): size / rate.
meets_objectives(rpo_hours, rto_hours, achieved_loss_hours, achieved_restore_hours): True if both achieved values are within their objectives.
restore_point(backups, deleted_at): backups is a sorted list of backup times (hours); return the latest backup taken at or before deleted_at (or None).
replica_has_row(replica_lag_s, deleted_at, now): a replica applies the delete after `replica_lag_s`; return True while the row still exists on the replica (now < deleted_at + lag).
backup_verified(job_completed, restore_tested, checksum_ok): a backup counts as usable only if a restore was actually tested AND the checksum matched (a completed job alone proves nothing).

def worst_case_loss_hours(backup_interval_hours, log_interval_minutes=None):
    pass

def restore_seconds(size_bytes, bytes_per_second):
    pass

def meets_objectives(rpo_hours, rto_hours, achieved_loss_hours, achieved_restore_hours):
    pass

def restore_point(backups, deleted_at):
    pass

def replica_has_row(replica_lag_s, deleted_at, now):
    pass

def backup_verified(job_completed, restore_tested, checksum_ok):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does a replica protect against every accidental logical deletion?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a time-based target A over W minutes, allowed bad time is $(1-A)W$ . A simplified two-replica failover availability model is $1-(1-a)^2$ only if failures are independent and failover is perfect. It excludes many shared-dependency and detection failures.

Worked example

A 30-day time window contains 43,200 minutes. At 99.9% time-based availability, allowed bad time is 43.2 minutes; at 99.99%, it is 4.32 minutes. Two independent 99%-available replicas with ideal failover give 99.99% in the toy model, not a general deployment guarantee.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
minutes=30*24*60
assert minutes == 43200
assert isclose((1-.999)*minutes,43.2)
assert isclose((1-.9999)*minutes,4.32)
assert isclose(1-(1-.99)**2,.9999)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Availability and durability differ. An average is not a latency percentile. Backup existence is not proof of recoverability. Correlated failures can dominate replica calculations.