SD10.2 · LESSON 2 OF 6

Error budgets

Define service objectives and use telemetry, failure handling and recovery procedures to protect user outcomes.

THE IDEA

Error budgets

The allowed bad fraction is an error budget. At 99.9% over 30 days it is 43.2 minutes: a 20-minute outage spends 46% of it. The burn rate compares the current error ratio with the allowed one: 0.5% errors against a 0.1% allowance burns five times too fast, exhausting the month in six days. Budgets turn reliability into a trade-off you can spend on releases, and they only work if measurement and budget use the same event definition and window.

bad minutes so far (30-day window)20current error ratio (%)0.5

budget at 99.9%: 43.2 minutes used: 20 (46.3%) remaining: 23.2 burn rate = 0.5% / 0.1% = 5 → a 30-day budget lasts 6 days alert: open a ticket budget left to spend on change and experiments

A 99.9% target does not forbid failure: it defines how much is allowed, and that allowance can be spent on releases and experiments. Measure and budget with the same event definition and window.

Spend minutes and change the error ratio.

Predict first. Does a 99.9% target mean no failures are allowed?

Chapter context

Reliability is measured against what users need, not whether every component reports healthy. Observability connects symptoms to evidence, while resilience and recovery determine what happens when dependencies or infrastructure fail.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute budget use and burn rate by hand, then write the budget and alert logic.

bad minutes so far (30-day window)20current error ratio (%)0.5

budget at 99.9%: 43.2 minutes used: 20 (46.3%) remaining: 23.2 burn rate = 0.5% / 0.1% = 5 → a 30-day budget lasts 6 days alert: open a ticket budget left to spend on change and experiments

A 99.9% target does not forbid failure: it defines how much is allowed, and that allowance can be spent on releases and experiments. Measure and budget with the same event definition and window.

fraction used by a 20-minute outage burn rate at 0.5% errors days the budget lasts 99.9% means no failures allowed? (yes/no)

To do: The monthly budget at 99.9% is 43.2 bad minutes. A 20-minute full outage uses what fraction of it (3 decimals)?
To do: Burn rate = observed error ratio / allowed error ratio. The error ratio is 0.5% (0.005) and the SLO allows 0.1% (0.001). Burn rate?
To do: At that burn rate, a 30-day budget lasts how many days?
To do: Does a 99.9% target mean no failures are allowed? Type yes or no

Part 2. Do it in code

budget_minutes(slo, days) and consumed_fraction(bad_minutes, slo, days): the share of the time budget used.
remaining_minutes(bad_minutes, slo, days): the unused budget (negative once overspent).
burn_rate(error_ratio, slo): error_ratio / (1 − slo).
days_to_exhaust(burn, window_days): window_days / burn (infinity when the burn is 0).
alert_level(burn): "page" if burn >= 10, "ticket" if burn >= 2, otherwise "ok".
freeze_releases(consumed): True once the budget is fully spent (consumed >= 1).

def budget_minutes(slo, days):
    pass

def consumed_fraction(bad_minutes, slo, days):
    pass

def remaining_minutes(bad_minutes, slo, days):
    pass

def burn_rate(error_ratio, slo):
    pass

def days_to_exhaust(burn, window_days):
    pass

def alert_level(burn):
    pass

def freeze_releases(consumed):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does a 99.9% target mean no failures are allowed?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a time-based target A over W minutes, allowed bad time is $(1-A)W$ . A simplified two-replica failover availability model is $1-(1-a)^2$ only if failures are independent and failover is perfect. It excludes many shared-dependency and detection failures.

Worked example

A 30-day time window contains 43,200 minutes. At 99.9% time-based availability, allowed bad time is 43.2 minutes; at 99.99%, it is 4.32 minutes. Two independent 99%-available replicas with ideal failover give 99.99% in the toy model, not a general deployment guarantee.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
minutes=30*24*60
assert minutes == 43200
assert isclose((1-.999)*minutes,43.2)
assert isclose((1-.9999)*minutes,4.32)
assert isclose(1-(1-.99)**2,.9999)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Availability and durability differ. An average is not a latency percentile. Backup existence is not proof of recoverability. Correlated failures can dominate replica calculations.