L3.4 · LESSON 4 OF 6

Log loss

Build a smoothed bigram model and connect next-token probabilities to log loss and perplexity.

THE IDEA

Scoring a model: log loss and perplexity

To judge a model look at the probability it assigned to what actually happened. The log loss of one prediction is −ln p: 0 for a certain correct guess, larger the less probable the truth was judged. Average it over a text. Perplexity is exp(average log loss); it reads as "as uncertain as picking evenly among this many tokens".

The worked example: with smoothing, P(b | a) = 3/4, so each alternating step costs −ln 0.75 ≈ 0.2877 nats and the perplexity is 4/3.

Four predictions for the sequence a b a b a. Each slider is the probability the model gave to the token that actually came next (set it to 0 to see the problem).

token 10.75token 20.75token 30.75token 40.75

token	probability	−ln p
1	0.75	0.2877
2	0.75	0.2877
3	0.75	0.2877
4	0.75	0.2877

average log loss = 0.2877 nats perplexity = exp(loss) = 1.3333 ("as unsure as choosing among 1.33 equally likely tokens")

Set the probability the model gave to each true next token. Try 0.

Predict first. Is a zero-probability correct token harmless if every other prediction is good?

Chapter context

A causal language model predicts the next token from the allowed prefix. A bigram model is a transparent starting point: it uses only the immediately preceding token, making counts, uncertainty and sampling easy to inspect.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute log loss and perplexity by hand, then write them and a held-out evaluation in Python.

Four predictions for the sequence a b a b a. Each slider is the probability the model gave to the token that actually came next (set it to 0 to see the problem).

token 10.75token 20.75token 30.75token 40.75

token	probability	−ln p
1	0.75	0.2877
2	0.75	0.2877
3	0.75	0.2877
4	0.75	0.2877

average log loss = 0.2877 nats perplexity = exp(loss) = 1.3333 ("as unsure as choosing among 1.33 equally likely tokens")

−ln(0.75), 3 decimals perplexity when every token gets 0.25 perplexity for 0.5 and 0.25 (3 decimals)

To do: The model gave the true next token probability 0.75. Log loss −ln(0.75) in nats (3 decimals)
To do: Perplexity is exp(average log loss). If the true token always gets probability 0.25, the perplexity is…
To do: Two tokens were predicted with probabilities 0.5 and 0.25. Their perplexity, 3 decimals

Part 2. Do it in code

nll(probs): the average negative natural log of the probabilities of the true tokens. If any probability is 0 return float('inf').
perplexity(probs): exp of nll(probs).
sequence_probs(tokens, P, vocab): the probability the model P (rows = previous token) gave to each actual next token, for all adjacent pairs of tokens.
evaluate(train, test, vocab, alpha=1.0): count bigrams in train, smooth each row with alpha, and return (nll, perplexity) on the pairs of test. With alpha = 0 an unseen test pair gives inf.

import numpy as np

def nll(probs):
    pass

def perplexity(probs):
    pass

def sequence_probs(tokens, P, vocab):
    pass

def evaluate(train, test, vocab, alpha=1.0):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Is a zero-probability correct token harmless if every other prediction is good?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

P(y\mid x)=\frac{C(x,y)+\alpha}{\sum_j C(x,j)+\alpha V},\quad \operatorname{NLL}=-\frac1N\sum_t\log P(x_{t+1}\mid x_t).

V is the output vocabulary size and alpha is a nonnegative smoothing amount. Perplexity is exp(NLL) for natural-log loss.

Worked example

For ababa, a→b occurs twice and b→a occurs twice. With vocabulary {a,b} and alpha=1, P(b|a)=(2+1)/(2+2)=0.75. An alternating transition then incurs −ln(0.75)≈0.287682 nats, whose perplexity is 4/3.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log,exp,isclose
seq='ababa'
count=sum(a=='a' and b=='b' for a,b in zip(seq,seq[1:]))
assert count == 2
prob=(count+1)/(2+2)
assert prob == .75
assert isclose(-log(prob),.2876820724517809)
assert isclose(exp(-log(prob)),4/3)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Predicting the input token itself can create target leakage. Bigram counts depend on token boundaries. Perplexities across incompatible tokenizers are not directly comparable.