E4.6 · LESSON 6 OF 6

Service objectives

Trace static and iteration-level schedules and explain chunked prefill, preemption and latency tradeoffs.

THE IDEA

Judging a schedule by service objectives

A schedule is good or bad relative to a goal. State the goal as an objective such as "95% of requests finish within 0.5 s" and measure it with percentiles and attainment under a stated arrival process (how requests really arrive, in bursts or evenly). A mean can look fine while the tail is bad, and a fixed batch that happens to suit your test says little about real traffic.

latency objective (s)0.5

0.3

0.33

0.35

0.38

0.4

0.42

0.45

0.5

0.9

1.2

attainment (≤ 0.5 s): 8 of 10 = 0.8 mean 0.523 s median 0.41 s p90 0.93 s p99 1.173 s

A schedule can keep the hardware 100% busy and still miss an interactive target, and a good average can hide a slow tail. Judge a policy with tail latency and completion rates under a stated arrival process, not on one favourable fixed batch.

Ten request latencies. Move the objective and read attainment, mean and tail.

Predict first. Can a schedule with higher utilisation still miss an interactive latency target?

Chapter context

Requests arrive at different times and finish after different output lengths. Scheduling decides which work enters each iteration and how waiting prompts compete with ongoing generation.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute attainment, a percentile and the mean for ten latencies, then write the report and a target check.

latency objective (s)0.5

0.3

0.33

0.35

0.38

0.4

0.42

0.45

0.5

0.9

1.2

attainment (≤ 0.5 s): 8 of 10 = 0.8 mean 0.523 s median 0.41 s p90 0.93 s p99 1.173 s

fraction meeting 0.5 s 90th percentile (s) mean (s, 3 decimals)

To do: Ten request latencies (s): 0.30, 0.40, 0.35, 0.90, 0.50, 0.45, 0.38, 0.42, 1.20, 0.33. The objective is 0.5 s or less. What fraction of requests meet it?
To do: The 90th percentile (sorted, linear interpolation at position 0.9 × 9 = 8.1) in seconds?
To do: The mean latency (3 decimals)?

Part 2. Do it in code

mean(values) and percentile(values, q): the average, and the q-th percentile (0–100) with linear interpolation between sorted neighbours. ValueError for an empty list.
attainment(latencies, slo): the fraction of latencies that are at most slo.
slo_report(latencies, slo): a dict with attainment, mean, p50, p90 and p99.
meets_target(report, min_attainment, max_p99): True if report['attainment'] >= min_attainment and report['p99'] <= max_p99.
utilization_vs_latency(points): points is a list of (utilization, p95_latency) pairs. Return the point with the highest utilization whose p95 is at most 1.0; None if there is none.

def mean(values):
    pass

def percentile(values, q):
    pass

def attainment(latencies, slo):
    pass

def slo_report(latencies, slo):
    pass

def meets_target(report, min_attainment, max_p99):
    pass

def utilization_vs_latency(points):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can a schedule with higher utilization still miss an interactive latency target?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

In a simplified equal-cost decode schedule, utilization = active sequence-iterations / available sequence-iterations. This omits prefill costs and assumes uniform iteration time. A real scheduler needs measured compute, memory and latency constraints.

Worked example

Two slots serve requests A,B,C with output lengths 2,1,2. Static groups run A+B for two iterations, then C alone for two: 5 active slots out of 8 available. Continuous admission runs A+B, then A+C, then C: 5 active slots out of 6. Utilization rises from 62.5% to about 83.33% in this toy schedule; real token costs are not equal.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
static=[['A','B'],['A',None],['C',None],['C',None]]
continuous=[['A','B'],['A','C'],['C',None]]
util=lambda rows: sum(x is not None for row in rows for x in row)/(2*len(rows))
assert util(static) == .625
assert isclose(util(continuous),5/6)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Continuous batching does not guarantee a fixed improvement. Chunked prefill changes scheduling granularity, not the model's context dependencies. Include cache movement or recomputation costs when comparing preemption policies.