SD10.3 · LESSON 3 OF 6

Metrics, logs and traces

Define service objectives and use telemetry, failure handling and recovery procedures to protect user outcomes.

THE IDEA

Metrics, logs and traces

Metrics (rates and distributions) show symptoms, logs describe individual events, and traces follow one request across services. Choose summaries carefully: if 95 requests take 100 ms and 5 take 2,000 ms, the mean is a healthy-looking 195 ms while the 99th percentile is 2,000 ms. Keep metric labels bounded (20 endpoints × 5 status classes is 100 series; adding a million-valued user id makes it 100 million) and keep private payloads out of telemetry.

requests that are slow (% of 100)5slow latency (ms)2000

mean	p50	p95	p99
195 ms	100 ms	100 ms	2000 ms

the mean looks healthy while 5% of users wait 2000 ms: only the upper percentiles show it metrics: rates and distributions (symptoms) logs: individual events traces: one request across services

endpoint (20) × status class (5) = 100 time series never put private payloads in labels or logs

Change the slow tail, then add an unbounded label.

Predict first. Does a fleet-wide mean show the slowest users' experience?

Chapter context

Reliability is measured against what users need, not whether every component reports healthy. Observability connects symptoms to evidence, while resilience and recovery determine what happens when dependencies or infrastructure fail.

TRY IT YOURSELF

Part 1. Do it hands-on

Read a latency sample by hand, then write the summary, cardinality and redaction helpers.

requests that are slow (% of 100)5slow latency (ms)2000

mean	p50	p95	p99
195 ms	100 ms	100 ms	2000 ms

the mean looks healthy while 5% of users wait 2000 ms: only the upper percentiles show it metrics: rates and distributions (symptoms) logs: individual events traces: one request across services

endpoint (20) × status class (5) = 100 time series never put private payloads in labels or logs

mean latency (95 × 100, 5 × 2000) p99 series, 20 × 5 labels series with user_id the mean shows the slowest users? (yes/no)

To do: 100 requests: 95 take 100 ms and 5 take 2000 ms. The mean latency (ms)?
To do: The 99th percentile (nearest-rank: the 99th smallest of 100 values), in ms?
To do: A metric has labels endpoint (20 values) and status class (5 values). Time series it creates?
To do: Someone adds a user_id label with 1,000,000 distinct values. Time series now?
To do: Does a fleet-wide mean show the slowest users' experience? Type yes or no

Part 2. Do it in code

mean(xs): the average. percentile(xs, p): nearest-rank — sort, take the value at rank ceil(p/100 · n) (1-based); p outside (0, 100] raises ValueError; an empty list raises ValueError.
summary(xs): {"mean", "p50", "p95", "p99"}.
series_count(label_sizes): the number of time series a metric creates: the product of the number of distinct values of each label.
bounded(label_sizes, limit): True if series_count <= limit.
redact(event, private_keys): a new dict with the values of private keys replaced by "[redacted]" (do not mutate the input).
slowest_share(xs, threshold): the fraction of values above the threshold.

import math

def mean(xs):
    pass

def percentile(xs, p):
    pass

def summary(xs):
    pass

def series_count(label_sizes):
    pass

def bounded(label_sizes, limit):
    pass

def redact(event, private_keys):
    pass

def slowest_share(xs, threshold):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does a fleet-wide mean show the slowest users' experience?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a time-based target A over W minutes, allowed bad time is $(1-A)W$ . A simplified two-replica failover availability model is $1-(1-a)^2$ only if failures are independent and failover is perfect. It excludes many shared-dependency and detection failures.

Worked example

A 30-day time window contains 43,200 minutes. At 99.9% time-based availability, allowed bad time is 43.2 minutes; at 99.99%, it is 4.32 minutes. Two independent 99%-available replicas with ideal failover give 99.99% in the toy model, not a general deployment guarantee.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
minutes=30*24*60
assert minutes == 43200
assert isclose((1-.999)*minutes,43.2)
assert isclose((1-.9999)*minutes,4.32)
assert isclose(1-(1-.99)**2,.9999)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Availability and durability differ. An average is not a latency percentile. Backup existence is not proof of recoverability. Correlated failures can dominate replica calculations.