SD10.4 · LESSON 4 OF 6

Timeouts, backoff and isolation

Define service objectives and use telemetry, failure handling and recovery procedures to protect user outcomes.

THE IDEA

Timeouts, backoff and isolation

A failing dependency should not consume every resource you have. Timeouts bound waiting. Retries need limits: three layers each trying three times turn one request into 27 calls to the bottom service, which can finish off a service that was merely overloaded. Add jittered backoff so clients do not retry in lockstep, a circuit breaker that stops calling a dependency that keeps failing, and bulkheads that cap how many workers any one dependency can hold. These are different policies for different problems.

layers that retry3attempts per layer3

one user request can become 3 ^ 3 = 27 calls to the bottom service when everything fails fix: bounded retries at ONE layer, exponential backoff with jitter, deadlines

Event 1 of 7

call 1: dependency fails (consecutive failures 1)

bulkheads: a pool of 30 workers split 20 for A and 10 for B. If A hangs it can tie up at most its own 20; B keeps 10. breakers stop calling a failing dependency; bulkheads limit how much damage it can do to everything else

Change the layers and attempts, then step through the breaker.

Predict first. Can aggressive retries make an overloaded service worse?

Chapter context

Reliability is measured against what users need, not whether every component reports healthy. Observability connects symptoms to evidence, while resilience and recovery determine what happens when dependencies or infrastructure fail.

TRY IT YOURSELF

Part 1. Do it hands-on

Count amplified calls by hand, then write backoff, a circuit breaker and a bulkhead.

layers that retry3attempts per layer3

one user request can become 3 ^ 3 = 27 calls to the bottom service when everything fails fix: bounded retries at ONE layer, exponential backoff with jitter, deadlines

Event 1 of 7

call 1: dependency fails (consecutive failures 1)

calls from 3 layers × 3 attempts calls reaching the dependency while the breaker is open workers left for B aggressive retries can worsen overload? (yes/no)

To do: Three service layers each retry a failing call up to 3 times in total. One user request can cause how many calls to the bottom service?
To do: A circuit breaker opens after 5 consecutive failures and then fails calls immediately. After 5 failures, how many further calls reach the failing dependency while it is open?
To do: A worker pool of 30 is split into bulkheads: 20 for dependency A, 10 for dependency B. A becomes slow and ties up all of its workers. Workers still available to B?
To do: Can aggressive retries make an overloaded service worse? Type yes or no

Part 2. Do it in code

retry_amplification(layers, attempts): attempts ** layers calls to the bottom service for one user request.
backoff_with_jitter(base, attempt, cap, u): the wait before retry number `attempt` (1-based): min(cap, base · 2 ** (attempt − 1)) × u, where u in [0, 1] is the jitter factor passed in ("full jitter").
class CircuitBreaker(threshold, open_for): states "closed", "open", "half_open". call(ok_fn, now): in closed state call ok_fn() — a False result or an exception counts as a failure; `threshold` CONSECUTIVE failures open the breaker (record opened_at = now). While open and now − opened_at < open_for, return "rejected" WITHOUT calling ok_fn. After open_for elapses the next call is a half_open trial: success closes the breaker (failure counter reset), failure re-opens it. Return "ok" or "failed" for executed calls.
class Bulkhead(sizes): sizes maps a dependency to its worker limit. acquire(dep) → True if a worker is free (and takes it), release(dep), free(dep) → remaining workers.

def retry_amplification(layers, attempts):
    pass

def backoff_with_jitter(base, attempt, cap, u):
    pass

class CircuitBreaker:
    def __init__(self, threshold, open_for):
        self.threshold = threshold
        self.open_for = open_for
        self.failures = 0
        self.state = "closed"
        self.opened_at = None

    def call(self, ok_fn, now):
        pass

class Bulkhead:
    def __init__(self, sizes):
        self.limit = dict(sizes)
        self.used = {k: 0 for k in sizes}

    def acquire(self, dep):
        pass

    def release(self, dep):
        pass

    def free(self, dep):
        pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can aggressive retries make an overloaded service worse?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a time-based target A over W minutes, allowed bad time is $(1-A)W$ . A simplified two-replica failover availability model is $1-(1-a)^2$ only if failures are independent and failover is perfect. It excludes many shared-dependency and detection failures.

Worked example

A 30-day time window contains 43,200 minutes. At 99.9% time-based availability, allowed bad time is 43.2 minutes; at 99.99%, it is 4.32 minutes. Two independent 99%-available replicas with ideal failover give 99.99% in the toy model, not a general deployment guarantee.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
minutes=30*24*60
assert minutes == 43200
assert isclose((1-.999)*minutes,43.2)
assert isclose((1-.9999)*minutes,4.32)
assert isclose(1-(1-.99)**2,.9999)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Availability and durability differ. An average is not a latency percentile. Backup existence is not proof of recoverability. Correlated failures can dominate replica calculations.