E9.4 · LESSON 4 OF 6

Autoscaling

Calculate cost per token, estimate stable concurrency and define operational metrics tied to user-facing objectives.

THE IDEA

Autoscaling is late by design

Adding a replica means finding a machine, loading the weights and warming up, so it helps only after a delay. Decisions should use queue and arrival signals and count replicas that are already starting. Removing replicas is risky too: in-flight requests and cached state are lost and a quick re-scale pays the whole startup again, so remove slowly and only when idle.

weights to load (GB)14load speed (GB/s)2warm-up (s)5arrivals per second12capacity now (per s)8

a new replica serves after 14/2 + 5 = 12 s while you wait, the backlog grows by (12 − 8) × 12 = 48 requests replicas needed for 12/s at 8/s each: 2

Autoscaling reacts late: loading weights takes real time. Use queue length and arrival signals, count replicas that are already starting so you do not order twice, and scale down slowly (one at a time, only when idle) because reloading is expensive and in-flight requests need to finish.

Change the model size, load speed and demand.

Predict first. Does a newly requested replica necessarily serve immediately?

Chapter context

An efficient service delivers useful requests within latency and quality targets at a sustainable cost. Hardware utilization, application traffic and failure handling determine whether an attractive benchmark becomes an economical deployment.

TRY IT YOURSELF

Part 1. Do it hands-on

Work out the cost of waiting for new capacity, then write a small scaling policy.

weights to load (GB)14load speed (GB/s)2warm-up (s)5arrivals per second12capacity now (per s)8

a new replica serves after 14/2 + 5 = 12 s while you wait, the backlog grows by (12 − 8) × 12 = 48 requests replicas needed for 12/s at 8/s each: 2

startup seconds (14 GB, 2 GB/s, 5 s) requests piled up in 12 s (12/s vs 8/s) replicas for 12/s at 8/s each

To do: A new replica must load 14 GB of weights at 2 GB/s and then warm up for 5 s. Seconds before it can serve?
To do: Demand jumps to 12 requests/s while current capacity is 8/s. During the 12 s startup delay, how many requests pile up?
To do: Each replica serves 8 requests/s. To handle 12/s you need ceil(12/8) replicas. How many?

Part 2. Do it in code

startup_seconds(weight_bytes, load_bytes_per_s, warmup_s): load time plus warm-up.
backlog_during_startup(arrival_rate, capacity, startup_s): requests that pile up while waiting for new capacity: max(0, arrival − capacity) · startup.
replicas_for(arrival_rate, per_replica_rate, headroom=0.0): ceil(arrival · (1 + headroom) / per_replica_rate); at least 1.
decide(queue_len, ready, starting, arrival_rate, per_replica_rate, min_replicas=1, max_replicas=10): the change in replicas to start (a non-negative number to add) or a negative number to remove. Rules: needed = replicas_for(arrival_rate, per_replica_rate); counting replicas that are already starting; if needed > ready + starting add the difference (never beyond max_replicas); if needed < ready and queue_len == 0 remove ONE replica at a time (never below min_replicas) and never remove while any replica is still starting; otherwise 0.

import math

def startup_seconds(weight_bytes, load_bytes_per_s, warmup_s):
    pass

def backlog_during_startup(arrival_rate, capacity, startup_s):
    pass

def replicas_for(arrival_rate, per_replica_rate, headroom=0.0):
    pass

def decide(queue_len, ready, starting, arrival_rate, per_replica_rate, min_replicas=1, max_replicas=10):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does a newly requested replica necessarily serve immediately?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For hourly cost H and useful sustained output rate r:

C_{million}=\frac{10^6H}{3600r}.

For a stable system under compatible averages, Little's law is $L=\lambda W$ : average in-system requests equal arrival rate times average time in system. It does not make an unstable queue stable.

Worked example

At a fictional $2/hour and 100 useful output tokens/s, cost is 2×1,000,000/(3600×100)≈$ 5.5556 per million output tokens, excluding other costs. If useful throughput falls to 50, cost doubles to about $11.1111. At four requests/s and two seconds average in-system time, average concurrent requests are eight.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
cost=lambda rate: 1_000_000*2/(3600*rate)
assert isclose(cost(100),5.555555555555555)
assert isclose(cost(50),2*cost(100))
assert 4*2 == 8

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Fictional arithmetic prices are not current provider quotes. Little's law requires a stable, consistently measured system. Counting failed or late tokens as useful throughput can hide poor service.