E9.6 · LESSON 6 OF 6

Quality in production

Calculate cost per token, estimate stable concurrency and define operational metrics tied to user-facing objectives.

THE IDEA

Quality is an operational metric too

A change that makes serving cheaper or faster (a smaller model, lower precision, a new prompt) can change the answers. Keep a task-specific evaluation set, run it on every change, and look at actual failures. Remember the set's size limits what you can conclude: a pass rate measured on 100 questions has a standard error near 3 percentage points, so a 7-point drop on 100 questions is not clearly real, while the same drop on 1000 questions is.

questions in the test set100old pass rate0.92new pass rate (cheaper setup)0.85

drop = 0.07 combined standard error = 0.0448 95% threshold = 1.96 × SE = 0.0879 the drop is within the noise of this test set: cannot be sure either way

Cutting infrastructure cost proves nothing about whether answers stayed useful. Keep a task-specific evaluation set, size it so that the changes you care about are detectable, and investigate observed failures even when a small test cannot confirm them statistically.

Change the test-set size and the pass rates.

Predict first. Does reducing infrastructure cost prove the answers remained useful?

Chapter context

An efficient service delivers useful requests within latency and quality targets at a sustainable cost. Hardware utilization, application traffic and failure handling determine whether an attractive benchmark becomes an economical deployment.

TRY IT YOURSELF

Part 1. Do it hands-on

Judge when a pass-rate drop is more than noise, then write the regression report.

questions in the test set100old pass rate0.92new pass rate (cheaper setup)0.85

drop = 0.07 combined standard error = 0.0448 95% threshold = 1.96 × SE = 0.0879 the drop is within the noise of this test set: cannot be sure either way

standard error of 90% on 100 questions 92→85 on 100 each: clearly real? (yes/no) on 1000 each? (yes/no)

To do: A task-specific test set of 100 questions shows 90% correct. The standard error of that pass rate is √(p(1−p)/n). (3 decimals)
To do: A cheaper setup scores 85/100 against the old 92/100. Using a 95% rule (drop > 1.96 × combined standard error), is that 7-point drop clearly real with only 100 questions each? Type yes or no
To do: The same pass rates measured on 1000 questions each (920/1000 versus 850/1000): is the drop clearly real? Type yes or no

Part 2. Do it in code

pass_rate(results): the fraction of True values in a list of pass/fail results. ValueError if empty.
std_error(p, n): sqrt(p(1−p)/n). n must be positive.
significant_drop(p_old, n_old, p_new, n_new, z=1.96): True if p_old − p_new is larger than z · sqrt(se_old² + se_new²).
regression_report(old_results, new_results): a dict with old_rate, new_rate, drop (old − new) and significant (using significant_drop).
needs_investigation(report, max_drop): True if the report shows a significant drop OR the raw drop is larger than max_drop (an alert threshold you chose for the task).

import math

def pass_rate(results):
    pass

def std_error(p, n):
    pass

def significant_drop(p_old, n_old, p_new, n_new, z=1.96):
    pass

def regression_report(old_results, new_results):
    pass

def needs_investigation(report, max_drop):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does reducing infrastructure cost prove the answers remained useful?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For hourly cost H and useful sustained output rate r:

C_{million}=\frac{10^6H}{3600r}.

For a stable system under compatible averages, Little's law is $L=\lambda W$ : average in-system requests equal arrival rate times average time in system. It does not make an unstable queue stable.

Worked example

At a fictional $2/hour and 100 useful output tokens/s, cost is 2×1,000,000/(3600×100)≈$ 5.5556 per million output tokens, excluding other costs. If useful throughput falls to 50, cost doubles to about $11.1111. At four requests/s and two seconds average in-system time, average concurrent requests are eight.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
cost=lambda rate: 1_000_000*2/(3600*rate)
assert isclose(cost(100),5.555555555555555)
assert isclose(cost(50),2*cost(100))
assert 4*2 == 8

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Fictional arithmetic prices are not current provider quotes. Little's law requires a stable, consistently measured system. Counting failed or late tokens as useful throughput can hide poor service.