Application state
Route requests across instances and calculate capacity with headroom, startup time and instance failures.
THE IDEA
Application state
A stateless request handler keeps nothing between requests, so any instance can serve any request and one can fail without losing user data; the state lives in a shared or durable store. Sticky routing keeps a user on one instance with its in-memory state, which saves a lookup but loses that instance's sessions when it dies (300 of 900 here) and can leave the fleet unbalanced.
Stateless means the request handler keeps no local state between requests, not that the product stores nothing: the data lives in a shared or durable store the handlers use.
Predict first. Does calling a service stateless mean the product stores no data?
Chapter context
Load balancing distributes work among eligible backends. Scaling changes available capacity. Both require the application to manage state and dependencies so adding a process actually adds useful throughput.
TRY IT YOURSELF
Part 1. Do it hands-on
Count lost sessions by hand, then write sticky routing, failure re-routing and the costs.
Stateless means the request handler keeps no local state between requests, not that the product stores nothing: the data lives in a shared or durable store the handlers use.
- To do: 900 user sessions are held in memory on 3 instances, sticky-routed evenly (300 each). One instance crashes. Sessions lost?
- To do: With sessions in an external shared store, sessions lost when an instance crashes?
- To do: The price: each request now reads its session from the store. Extra round trips per request (one read)?
- To do: Does calling a service stateless mean the product stores no data? Type yes or no
Part 2. Do it in code
- sessions_lost(n_sessions, n_instances, sticky): sessions lost when ONE instance crashes. Sticky with in-memory sessions spread evenly loses n_sessions // n_instances; sessions in an external store lose 0.
- home(session_id, n): the instance a sticky router would send the session to: sum of character codes of the id modulo n.
- reroute(session_ids, n, failed): the new instance for each id when instance `failed` is down: sessions whose home is alive keep it; the others go to the next alive instance (failed + 1) % n. Return a dict id → instance.
- moved(session_ids, n, failed): how many sessions changed instance in reroute.
- extra_lookups(external_store): 1 read per request when sessions live in an external store, else 0.
- request_latency(base_ms, store_ms, external_store): base_ms plus the store round trip if the store is external.
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Does calling a service stateless mean the product stores no data?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
With demand R, per-instance capacity c and intended usable fraction u, require . To keep that same target after one failure, require . These are rate constraints, not a complete queueing or availability model.
Worked example
Demand is 900 requests/s. Each instance sustains 400 requests/s, but the target usable fraction is 75%, or 300 requests/s. Three instances meet normal demand. Four are required to maintain the same target capacity after losing one: (4−1)×300=900.
Reference code
The following example uses the Python standard library and does not call a remote service.
from math import ceil
demand,capacity,usable=900,400,.75
normal=ceil(demand/(capacity*usable))
resilient=normal+1
assert normal == 3 and resilient == 4
assert (resilient-1)*capacity*usable == demand
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
Horizontal scaling can move the bottleneck to a database. Healthy and ready are not identical. A load balancer does not repair session or transaction state lost with a backend.