L3.2 · LESSON 2 OF 6

Counting bigrams

Build a smoothed bigram model and connect next-token probabilities to log loss and perplexity.

THE IDEA

The simplest language model is a table of counts

A bigram model predicts the next token from only the current one. Build it by counting: for every pair, add one to the cell (previous token, next token). Dividing a row by its sum turns the counts into probabilities of what follows that token.

letters

previous ↓ / next →	a	b	row sum
a	0	2	2
b	2	0	2

5 letters → 4 transitions; the row sums add up to 4 The last letter has no successor, so its own row can be empty.

Type letters and read the counts. The row sums tell you how many times each token was followed by something.

Predict first. In ababa, how often does a→b occur?

Python

for a, b in zip(tokens, tokens[1:]):
    C[idx[a], idx[b]] += 1

Chapter context

A causal language model predicts the next token from the allowed prefix. A bigram model is a transparent starting point: it uses only the immediately preceding token, making counts, uncertainty and sampling easy to inspect.

TRY IT YOURSELF

Part 1. Do it hands-on

Count transitions by hand, then build the count matrix and the probability table in Python.

letters

previous ↓ / next →	a	b	row sum
a	0	2	2
b	2	0	2

5 letters → 4 transitions; the row sums add up to 4 The last letter has no successor, so its own row can be empty.

a→b in a b a b a row sum of a in a a b a b distinct pairs in a b c a b

To do: In the sequence a b a b a, how often does a → b occur?
To do: In a a b a b: the row sum for "a" (how many times a is followed by anything)
To do: In a b c a b: how many different (previous, next) pairs occur at least once?

Part 2. Do it in code

count_matrix(tokens, vocab): an array (V, V) where entry [i, j] counts how often vocab[j] directly follows vocab[i]. Raise ValueError if a token is not in the vocabulary.
row_sums(C): the number of observed transitions out of each token.
unsmoothed_probs(C): divide each row by its row sum. A row with sum 0 stays all zeros.
most_likely_next(C, vocab, token): the vocabulary entry with the highest count after token (the first one if tied); None if the token was never followed by anything.

import numpy as np

def count_matrix(tokens, vocab):
    pass

def row_sums(C):
    pass

def unsmoothed_probs(C):
    pass

def most_likely_next(C, vocab, token):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

In ababa, how often does a→b occur?

ATwice: the pairs are ab, ba, ab, ba, so ab occurs twice (and so does ba).BOnce: only the first ab is counted.CThree times: every b is preceded by an a.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

P(y\mid x)=\frac{C(x,y)+\alpha}{\sum_j C(x,j)+\alpha V},\quad \operatorname{NLL}=-\frac1N\sum_t\log P(x_{t+1}\mid x_t).

V is the output vocabulary size and alpha is a nonnegative smoothing amount. Perplexity is exp(NLL) for natural-log loss.

Worked example

For ababa, a→b occurs twice and b→a occurs twice. With vocabulary {a,b} and alpha=1, P(b|a)=(2+1)/(2+2)=0.75. An alternating transition then incurs −ln(0.75)≈0.287682 nats, whose perplexity is 4/3.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log,exp,isclose
seq='ababa'
count=sum(a=='a' and b=='b' for a,b in zip(seq,seq[1:]))
assert count == 2
prob=(count+1)/(2+2)
assert prob == .75
assert isclose(-log(prob),.2876820724517809)
assert isclose(exp(-log(prob)),4/3)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Predicting the input token itself can create target leakage. Bigram counts depend on token boundaries. Perplexities across incompatible tokenizers are not directly comparable.