SD6.3 · LESSON 3 OF 6

Health and readiness

Route requests across instances and calculate capacity with headroom, startup time and instance failures.

THE IDEA

Health and readiness

Two different questions. Liveness: is the process stuck, so that it needs a restart? Readiness: should it receive traffic right now? A process still loading its model is alive but not ready. Health checks have their own failure modes: they can be stale, and a readiness check that requires a shared dependency turns that dependency's outage into a total outage, because every instance reports unready at once.

instance	alive (liveness)	model loaded	ready	gets traffic
#1	yes	yes	yes	yes
#2	yes	yes	yes	yes
#3	yes	no	no	no
#4	NO → restart	no	no	no

instances receiving traffic: 2 needing a restart: 1

Liveness asks whether the process needs recovery (restart it). Readiness asks whether it should receive traffic (not while loading). Checks can be stale, and overly strict ones can take the whole fleet out when a shared dependency fails.

Take the database down and make readiness strict.

Predict first. Should a process still loading its required model always receive normal traffic?

Chapter context

Load balancing distributes work among eligible backends. Scaling changes available capacity. Both require the application to manage state and dependencies so adding a process actually adds useful throughput.

TRY IT YOURSELF

Part 1. Do it hands-on

Decide traffic and restarts by hand, then write liveness, readiness and a fail-open fallback.

instance	alive (liveness)	model loaded	ready	gets traffic
#1	yes	yes	yes	yes
#2	yes	yes	yes	yes
#3	yes	no	no	no
#4	NO → restart	no	no	no

instances receiving traffic: 2 needing a restart: 1

instances receiving traffic (normal) instances needing a restart receiving traffic when readiness requires a downed DB normal traffic to a loading process? (yes/no)

To do: Four instances: #1 and #2 are healthy and loaded; #3 is alive but still loading its model; #4 has deadlocked (not alive). Readiness decides who gets traffic. How many instances receive traffic?
To do: Liveness decides who needs recovery (a restart). How many instances need one?
To do: Now the shared database goes down. If every instance's readiness check requires the database, how many instances receive traffic?
To do: Should a process still loading its required model receive normal traffic? Type yes or no

Part 2. Do it in code

An instance is a dict {"id", "alive", "loaded", "db_ok"}.
liveness(i): True if the process is alive (it does not need recovery).
readiness(i, check_db=False): alive AND loaded AND (db_ok if check_db).
targets(instances, check_db=False): the ids that should receive traffic.
to_restart(instances): the ids that are not alive.
route_with_panic(instances, check_db=False): like targets, but if NO instance is ready, fail open: return the ids of the alive instances (a documented trade-off to avoid a total outage).
stale_check_ok(last_ok_age_s, max_age_s): a health result is trusted only if it is not older than max_age_s.

def liveness(i):
    pass

def readiness(i, check_db=False):
    pass

def targets(instances, check_db=False):
    pass

def to_restart(instances):
    pass

def route_with_panic(instances, check_db=False):
    pass

def stale_check_ok(last_ok_age_s, max_age_s):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Should a process still loading its required model always receive normal traffic?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

With demand R, per-instance capacity c and intended usable fraction u, require $ncu\ge R$ . To keep that same target after one failure, require $(n-1)cu\ge R$ . These are rate constraints, not a complete queueing or availability model.

Worked example

Demand is 900 requests/s. Each instance sustains 400 requests/s, but the target usable fraction is 75%, or 300 requests/s. Three instances meet normal demand. Four are required to maintain the same target capacity after losing one: (4−1)×300=900.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import ceil
demand,capacity,usable=900,400,.75
normal=ceil(demand/(capacity*usable))
resilient=normal+1
assert normal == 3 and resilient == 4
assert (resilient-1)*capacity*usable == demand

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Horizontal scaling can move the bottleneck to a database. Healthy and ready are not identical. A load balancer does not repair session or transaction state lost with a backend.