SD10.6 · LESSON 6 OF 6

Incidents and learning

Define service objectives and use telemetry, failure handling and recovery procedures to protect user outcomes.

THE IDEA

Incidents and learning

Redundancy math is a toy until you check independence. Two 99%-available replicas with independent failures and perfect failover give 99.99%; but if both depend on a shared 99.9% service the total is about 99.89%, and if they always fail together you have 99% and no benefit. During an incident, preserve a timeline, the evidence and the customer impact; afterwards improve detection and recovery instead of only silencing the alert.

availability of one replica0.99replicas2shared dependency availability0.999

independent failures, perfect failover: 1 − (1 − 0.99)^2 = 0.9999 with the shared dependency in series: 0.9989 (the dependency caps the whole thing) if the replicas always fail together: 0.99 (no benefit at all)

The independent-failure formula is a toy: shared racks, shared configuration, shared bugs and imperfect detection make failures correlated. After an incident, preserve a timeline, the evidence and the customer impact, then improve detection and recovery rather than assuming the redundancy worked.

Vary availability, replicas and the shared dependency.

Predict first. Does adding two replicas guarantee their failures are independent?

Chapter context

Reliability is measured against what users need, not whether every component reports healthy. Observability connects symptoms to evidence, while resilience and recovery determine what happens when dependencies or infrastructure fail.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute availability with and without correlation by hand, then write the models and incident helpers.

availability of one replica0.99replicas2shared dependency availability0.999

two independent 99% replicas with a shared 99.9% dependency fully correlated failures two replicas guarantee independence? (yes/no)

To do: Two replicas, each 99% available, failing independently with perfect failover: availability = 1 − (1 − 0.99)² (4 decimals)?
To do: Both replicas need one shared dependency that is 99.9% available (in series). Overall availability = 0.9999 × 0.999 (4 decimals)?
To do: If the two replicas always fail together (fully correlated), availability is just that of one replica. Value?
To do: Does adding two replicas guarantee their failures are independent? Type yes or no

Part 2. Do it in code

parallel(a, n): availability of n redundant components that fail independently with perfect failover: 1 − (1 − a)^n.
series(avails): the product of availabilities (all must work).
with_shared_dependency(a, n, dep): parallel(a, n) in series with a dependency of availability dep.
fully_correlated(a, n): n components that always fail together are no better than one: a.
mttr(incidents): incidents is a list of {"detected", "resolved"} times in minutes; the mean of resolved − detected (0 for an empty list).
timeline(events): events is a list of {"t", "what"}; return the "what" strings sorted by t (ties keep input order).
impact_minutes(start, end, fraction_affected): (end − start) × fraction_affected.

def parallel(a, n):
    pass

def series(avails):
    pass

def with_shared_dependency(a, n, dep):
    pass

def fully_correlated(a, n):
    pass

def mttr(incidents):
    pass

def timeline(events):
    pass

def impact_minutes(start, end, fraction_affected):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does adding two replicas guarantee their failures are independent?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a time-based target A over W minutes, allowed bad time is $(1-A)W$ . A simplified two-replica failover availability model is $1-(1-a)^2$ only if failures are independent and failover is perfect. It excludes many shared-dependency and detection failures.

Worked example

A 30-day time window contains 43,200 minutes. At 99.9% time-based availability, allowed bad time is 43.2 minutes; at 99.99%, it is 4.32 minutes. Two independent 99%-available replicas with ideal failover give 99.99% in the toy model, not a general deployment guarantee.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
minutes=30*24*60
assert minutes == 43200
assert isclose((1-.999)*minutes,43.2)
assert isclose((1-.9999)*minutes,4.32)
assert isclose(1-(1-.99)**2,.9999)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Availability and durability differ. An average is not a latency percentile. Backup existence is not proof of recoverability. Correlated failures can dominate replica calculations.