SD6.5 · LESSON 5 OF 6

Autoscaling delays

Route requests across instances and calculate capacity with headroom, startup time and instance failures.

THE IDEA

Autoscaling delays

An autoscaler has to notice the load, schedule an instance and wait for it to start: 30 + 20 + 40 = 90 seconds in the example. A burst from 500 to 2,000 requests a second against capacity for 500 leaves 1,500 req/s unserved for that whole time, 135,000 requests. Scale on a signal that reflects the real bottleneck, damp noisy samples so the fleet does not oscillate, and keep headroom for the lag.

scaling lag (seconds)3burst (req/s)2000

orange: demand blue dashed: usable capacity (instances of 250 req/s) the autoscaler sees the burst at t = 5 but new capacity only arrives at t = 8 requests above capacity in between: 4,500 realistic lag = observe 30 s + schedule 20 s + start 40 s = 90 s: at 1500 req/s over capacity that is 135,000 requests

Observation, scheduling and startup all take time, so capacity arrives after the burst. Use signals that reflect the real bottleneck, damp noisy measurements to avoid flapping, and keep headroom for the lag.

Change the lag and the size of the burst.

Predict first. Can a cold instance instantly absorb an already-arrived burst?

Chapter context

Load balancing distributes work among eligible backends. Scaling changes available capacity. Both require the application to manage state and dependencies so adding a process actually adds useful throughput.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute the lag and the shortfall by hand, then simulate an autoscaler with a delay.

scaling lag (seconds)3burst (req/s)2000

seconds from burst to usable capacity req/s over capacity (500 → 2000) requests over capacity during 90 s a cold instance absorbs the burst at once? (yes/no)

To do: Detecting the load takes 30 s, scheduling a new instance 20 s and its startup 40 s. Seconds from burst to usable capacity?
To do: Traffic jumps from 500 to 2000 requests/s; 2 instances (250 req/s each) are running. Requests per second above capacity?
To do: During the 90 s lag, how many requests exceed capacity (to be queued, slow or dropped)?
To do: Can a cold instance instantly absorb an already-arrived burst? Type yes or no

Part 2. Do it in code

scale_lag(observe_s, schedule_s, startup_s): the total delay from load change to usable capacity.
needed_instances(demand, capacity_per_instance, usable_fraction=1.0): ceil(demand / (capacity × usable)), at least 1.
capacity_series(demand, cap_per, initial_instances, lag): demand is a list of requests/s per second. Each second t the autoscaler computes the desired instances from demand[t]; they only become usable lag seconds later. For t < lag the capacity is initial_instances × cap_per; for t >= lag it is cap_per × needed_instances(demand[t − lag], cap_per). Return the list.
dropped(demand, capacity): the total requests above capacity (sum of max(0, demand − capacity)).
damped(history, window): the average of the last `window` measurements (avoid scaling on a single noisy sample); history must not be empty.

import math

def scale_lag(observe_s, schedule_s, startup_s):
    pass

def needed_instances(demand, capacity_per_instance, usable_fraction=1.0):
    pass

def capacity_series(demand, cap_per, initial_instances, lag):
    pass

def dropped(demand, capacity):
    pass

def damped(history, window):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can a cold instance instantly absorb an already-arrived burst?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

With demand R, per-instance capacity c and intended usable fraction u, require $ncu\ge R$ . To keep that same target after one failure, require $(n-1)cu\ge R$ . These are rate constraints, not a complete queueing or availability model.

Worked example

Demand is 900 requests/s. Each instance sustains 400 requests/s, but the target usable fraction is 75%, or 300 requests/s. Three instances meet normal demand. Four are required to maintain the same target capacity after losing one: (4−1)×300=900.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import ceil
demand,capacity,usable=900,400,.75
normal=ceil(demand/(capacity*usable))
resilient=normal+1
assert normal == 3 and resilient == 4
assert (resilient-1)*capacity*usable == demand

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Horizontal scaling can move the bottleneck to a database. Healthy and ready are not identical. A load balancer does not repair session or transaction state lost with a backend.