Horizontal and vertical growth
Route requests across instances and calculate capacity with headroom, startup time and instance failures.
THE IDEA
Horizontal and vertical growth
To carry more load you can give one node more resources (vertical) or add more nodes (horizontal). Horizontal growth only helps the part of the work that can be spread out. If 10% of every request is serialised, for example behind a shared lock or a single database, then 2 instances give a 1.818× speed-up, 10 give 5.263×, and no number of instances can ever exceed 1/0.1 = 10×.
| instances | 1 | 2 | 4 | 8 | 16 | 32 |
|---|---|---|---|---|---|---|
| speed-up | 1 | 1.82 | 3.08 | 4.71 | 6.4 | 7.8 |
A bigger instance adds resources to one node; more instances share the work. But shared storage, locks and serial steps do not shrink when you add replicas, so doubling the replicas does not double capacity.
Predict first. Does doubling application replicas always double end-to-end capacity?
Chapter context
Load balancing distributes work among eligible backends. Scaling changes available capacity. Both require the application to manage state and dependencies so adding a process actually adds useful throughput.
TRY IT YOURSELF
Part 1. Do it hands-on
Compute speed-ups by hand, then write the model and a search for the instances you need.
| instances | 1 | 2 | 4 | 8 | 16 | 32 |
|---|---|---|---|---|---|---|
| speed-up | 1 | 1.82 | 3.08 | 4.71 | 6.4 | 7.8 |
A bigger instance adds resources to one node; more instances share the work. But shared storage, locks and serial steps do not shrink when you add replicas, so doubling the replicas does not double capacity.
- To do: 10% of each request's work is serialised (a shared lock). Speed-up with n instances is 1 / (s + (1 − s)/n). With n = 2 (3 decimals)?
- To do: With n = 10 (3 decimals)?
- To do: However many instances you add, the speed-up can never exceed 1 / s. What is that limit here?
- To do: Does doubling application replicas always double end-to-end capacity? Type yes or no
Part 2. Do it in code
- speedup(n, serial): 1 / (serial + (1 − serial)/n); ValueError for n < 1 or serial outside [0, 1].
- efficiency(n, serial): speedup / n.
- speedup_limit(serial): 1 / serial (infinity when serial is 0).
- instances_for(target, serial): the smallest n whose speed-up is at least `target`, or None if the target is at or above the limit.
- capacity_with(n, small_capacity, serial): the effective requests/s of n small instances: speedup(n, serial) × small_capacity.
- instances_for_demand(demand, small_capacity, serial): the smallest n whose effective capacity reaches the demand, or None if no number of instances can.
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Does doubling application replicas always double end-to-end capacity?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
With demand R, per-instance capacity c and intended usable fraction u, require . To keep that same target after one failure, require . These are rate constraints, not a complete queueing or availability model.
Worked example
Demand is 900 requests/s. Each instance sustains 400 requests/s, but the target usable fraction is 75%, or 300 requests/s. Three instances meet normal demand. Four are required to maintain the same target capacity after losing one: (4−1)×300=900.
Reference code
The following example uses the Python standard library and does not call a remote service.
from math import ceil
demand,capacity,usable=900,400,.75
normal=ceil(demand/(capacity*usable))
resilient=normal+1
assert normal == 3 and resilient == 4
assert (resilient-1)*capacity*usable == demand
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
Horizontal scaling can move the bottleneck to a database. Healthy and ready are not identical. A load balancer does not repair session or transaction state lost with a backend.