Streaming and state

Four small rules recur through the chapter.

• Complete or cut off (F8.1): treat the reply as an answer only when the provider's stop reason means a natural stop (end_turn, stop, STOP), not a limit (max_tokens, length, MAX_TOKENS).
• Validation order (F8.2-F8.4): parse, then check fields, then check domain rules. Passing one stage says nothing about the next.
• Backoff (F8.5): wait $d_k=b\cdot 2^k\cdot j$ before retry $k$, where $b$ is the base delay and the jitter factor $j$ is a random number in $[0.5,1]$ (or $1$ without jitter). Retry only temporary errors, and stop after a fixed number of retries.
• Cost (F8.6): for prices $p_i$ and $p_o$ per million input and output tokens, and token counts $n_i$ and $n_o$:

With an output cap $m\ge n_o$, the worst-case output side for $N$ calls is $N\,m\,p_o/10^6$. The prices below are invented arithmetic inputs, not a product recommendation; use the provider's current price list for real budgeting.

One small helper per idea, all checked offline. A reply stopped with end_turn is usable but one stopped with max_tokens is not. The text {"sentiment": "neutral", "confidence": 0.7} passes all three validation stages, while a confidence of 7 or a reply with chatty text around the JSON is rejected. A 429 is retried but a 401 is not, and the delays for retries 0, 1 and 2 at a 0.5 s base are 0.5, 1.0 and 2.0 seconds. At illustrative prices of $2 and$6 per million tokens, a call with 1,000 input and 500 output tokens costs (1000×2 + 500×6)/1,000,000 = $0.005.

The following example uses the Python standard library and does not call a remote service. It is the same logic the labs let you try against a real model.

import json
from math import isclose

def usable(stop_reason):                      # F8.1: a limit means the text may be cut off
    return stop_reason in {"end_turn", "stop", "STOP"}

def validate(text):                           # F8.3: syntax, then fields, then rules
    data = json.loads(text)                   # raises ValueError on bad syntax
    if data.get("sentiment") not in {"positive", "negative", "neutral"}:
        raise ValueError("bad sentiment")
    if not 0 <= data.get("confidence", -1) <= 1:
        raise ValueError("confidence out of range")
    return data

def should_retry(status, attempt, max_retries=3):   # F8.5: temporary errors only, bounded
    return status in {429, 529} and attempt < max_retries

def delay(attempt, base=0.5, jitter=1.0):     # F8.5: exponential backoff
    return base * 2 ** attempt * jitter

def cost(n_in, n_out, p_in=2, p_out=6):       # F8.6: illustrative prices
    return (n_in * p_in + n_out * p_out) / 1_000_000

assert usable("end_turn") and not usable("max_tokens") and not usable("length")
assert validate('{"sentiment": "neutral", "confidence": 0.7}')["confidence"] == 0.7
for bad in ['{"sentiment": "neutral", "confidence": 7}', 'Sure! {"sentiment": "neutral"}']:
    try:
        validate(bad)
    except ValueError:
        pass
    else:
        raise AssertionError("accepted bad output")
assert should_retry(429, 0) and not should_retry(401, 0) and not should_retry(429, 3)
assert [delay(a) for a in range(3)] == [0.5, 1.0, 2.0]
assert isclose(cost(1000, 500), 0.005)
assert isclose(cost(1000, 500) * 10_000, 50)

JSON output can still be false or out of range. A model call is not required for every rule-based task. Never place a reusable private API credential in a public browser bundle: the labs here use a key you paste yourself, kept in your browser and sent only to the provider, and a shipped product would call the provider from a server instead. Reasoning models can spend part of the output limit on hidden reasoning, so a tiny limit may return empty text.