SD2.5 · LESSON 5 OF 6

Timeouts and retries

Trace a client request and design an HTTP contract with clear failure, retry and pagination behavior.

THE IDEA

Timeouts and retries

A timeout means the caller has no result. It does not mean the server did no work: a create may have committed just before the connection dropped. Retries therefore need rules: a bounded number of attempts, backoff between them, and a distinction between repeatable reads (safe to retry) and writes (which need explicit deduplication). Remember the cost: 3 attempts of up to 1 s with 100 ms and 200 ms waits can take 3.3 s, and a failing service may see three times its normal traffic.

attempts in total3timeout per attempt (ms)1000first backoff (ms)100requests per second1000

Attempt 1 of 3

attempt 1: sent, no answer within 1000 ms → timeout wait 100 ms, then retry the server may still have processed attempts 1…1

waiting between attempts: 100 + 200 = 300 ms worst case for the caller: 3 × 1000 + 300 = 3300 ms if the service is failing, 1000 req/s of clients with 3 attempts each can send up to 3000 req/s

A timeout means the caller has no result, not that the server did nothing. Bound the retries, add backoff (and jitter in practice), and retry only operations that are safe to repeat.

Vary attempts, timeout and backoff, then step through the attempts.

Predict first. Can a timed-out create request already have committed?

Chapter context

Clients communicate through name resolution, connections, transport security and application protocols. API semantics determine what a caller can safely infer or repeat when a network interruption hides an operation's outcome.

TRY IT YOURSELF

Part 1. Do it hands-on

Cost a retry policy by hand, then write the backoff and the retry loop.

attempts in total3timeout per attempt (ms)1000first backoff (ms)100requests per second1000

Attempt 1 of 3

attempt 1: sent, no answer within 1000 ms → timeout wait 100 ms, then retry the server may still have processed attempts 1…1

waiting between attempts: 100 + 200 = 300 ms worst case for the caller: 3 × 1000 + 300 = 3300 ms if the service is failing, 1000 req/s of clients with 3 attempts each can send up to 3000 req/s

A timeout means the caller has no result, not that the server did nothing. Bound the retries, add backoff (and jitter in practice), and retry only operations that are safe to repeat.

total wait, 3 attempts from 100 ms worst case with 1000 ms timeouts peak load from 1000 req/s a timed-out create may have committed? (yes/no)

To do: A client makes up to 3 attempts and waits 100 ms before the 2nd and 200 ms before the 3rd (exponential backoff from 100 ms). Total ms spent waiting between attempts?
To do: Each attempt may take its full 1000 ms timeout. Worst-case total ms for the call?
To do: A failing service receives 1000 requests/s and every client retries up to 3 attempts in total. Peak request rate it may see?
To do: Can a timed-out create request already have committed on the server? Type yes or no

Part 2. Do it in code

backoff_delays(attempts, base_ms, factor=2, cap_ms=None): the waits between attempts: attempts − 1 values base, base·factor, … each capped at cap_ms when given.
worst_case_ms(attempts, timeout_ms, base_ms, factor=2): every attempt hits its timeout, plus all the waits.
retry_load(rps, attempts): the peak request rate a failing service sees when every client uses all its attempts.
call_with_retry(fn, attempts, base_ms, sleeps): call fn(); if it raises TimeoutError wait (append the delay to the list sleeps — do not really sleep) and try again, up to `attempts` calls in total. Return fn's result; if every attempt times out raise TimeoutError. Other exceptions propagate immediately.
retryable(method, error): True only for a TimeoutError on an idempotent method (GET, PUT, DELETE, HEAD, OPTIONS).

def backoff_delays(attempts, base_ms, factor=2, cap_ms=None):
    pass

def worst_case_ms(attempts, timeout_ms, base_ms, factor=2):
    pass

def retry_load(rps, attempts):
    pass

def call_with_retry(fn, attempts, base_ms, sleeps):
    pass

def retryable(method, error):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can a timed-out create request already have committed?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For k sequential calls with round-trip latency r, the network component alone is approximately k*r under a simple fixed-RTT model. Independent parallel calls have an ideal max-path cost instead. This excludes connection setup, payload transfer, queueing and application computation.

Worked example

Three required sequential remote calls with 40 ms round trips consume about 120 ms of round-trip time before server work. For a separate write example, processing request key op-7 twice should create one logical result if the idempotency contract is implemented atomically. The calculation below is an offline model, not a distributed storage implementation.

Reference code

The following example uses the Python standard library and does not call a remote service.

assert 3*40 == 120
seen={}
def create(key,value):
    if key in seen:
        if seen[key] != value: raise ValueError('key reused with different intent')
        return seen[key]
    seen[key]=value
    return value
assert create('op-7','order-1') == create('op-7','order-1')
assert len(seen) == 1

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A timeout is not proof of rollback. HTTP transport security does not authorize object access. An in-memory deduplication dictionary does not supply durability or atomicity across servers.