L3.1 · LESSON 1 OF 6

Next-token targets

Build a smoothed bigram model and connect next-token probabilities to log loss and perplexity.

THE IDEA

What the model is asked to guess

A language model assigns a probability to the next token given what came before. Training data needs no labels: the text itself supplies them. Every token is the target for the token (or tokens) just before it, so the targets are the same sequence shifted left by one.

text add start and end markers

#	input (current token)		target (next token)
1	the	→	cat
2	cat	→	sat
3	sat	→	on
4	on	→	the
5	the	→	mat

6 tokens → 5 pairs inputs : the cat sat on the targets: cat sat on the mat (the same list shifted left by one)

Type any text. Each adjacent pair is one training example; start and end markers add one pair at each end.

Predict first. For tokens a, b, c, what are the adjacent training pairs?

Python

inputs, targets = tokens[:-1], tokens[1:]     # shifted by one

Chapter context

A causal language model predicts the next token from the allowed prefix. A bigram model is a transparent starting point: it uses only the immediately preceding token, making counts, uncertainty and sampling easy to inspect.

TRY IT YOURSELF

Part 1. Do it hands-on

Predict how many training pairs a sentence gives, then write the pair builders in Python.

text add start and end markers

#	input (current token)		target (next token)
1	the	→	cat
2	cat	→	sat
3	sat	→	on
4	on	→	the
5	the	→	mat

6 tokens → 5 pairs inputs : the cat sat on the targets: cat sat on the mat (the same list shifted left by one)

pairs from the 6-word sentence target when the input is "sat" pairs with start and end markers

To do: "the cat sat on the mat" has 6 tokens. How many (input → target) training pairs does it give?
To do: In that sentence, what is the TARGET when the input is "sat"? Type the word
To do: Add a start marker <s> in front and an end marker </s> at the end. Now how many pairs?

Part 2. Do it in code

make_pairs(tokens, bos=None, eos=None): the (input, target) pairs of adjacent tokens. If bos is given put it in front of the sequence first; if eos is given append it.
inputs_targets(tokens): return (inputs, targets) as two lists of equal length: tokens without the last one, and tokens without the first. Raise ValueError if there are fewer than 2 tokens.
window_pairs(tokens, context): pairs (tuple of `context` previous tokens, next token). Raise ValueError if context < 1. A sequence shorter than context + 1 gives no pairs.

def make_pairs(tokens, bos=None, eos=None):
    pass

def inputs_targets(tokens):
    pass

def window_pairs(tokens, context):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

For tokens a,b,c, what are the adjacent training pairs?

Aa→a, b→b and c→c, because each token predicts itself.Ba→b, b→c and c→a, because the sequence wraps around.Ca→b and b→c: the target is the next token, never the current one, so n tokens give n − 1 pairs.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

P(y\mid x)=\frac{C(x,y)+\alpha}{\sum_j C(x,j)+\alpha V},\quad \operatorname{NLL}=-\frac1N\sum_t\log P(x_{t+1}\mid x_t).

V is the output vocabulary size and alpha is a nonnegative smoothing amount. Perplexity is exp(NLL) for natural-log loss.

Worked example

For ababa, a→b occurs twice and b→a occurs twice. With vocabulary {a,b} and alpha=1, P(b|a)=(2+1)/(2+2)=0.75. An alternating transition then incurs −ln(0.75)≈0.287682 nats, whose perplexity is 4/3.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import log,exp,isclose
seq='ababa'
count=sum(a=='a' and b=='b' for a,b in zip(seq,seq[1:]))
assert count == 2
prob=(count+1)/(2+2)
assert prob == .75
assert isclose(-log(prob),.2876820724517809)
assert isclose(exp(-log(prob)),4/3)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Predicting the input token itself can create target leakage. Bigram counts depend on token boundaries. Perplexities across incompatible tokenizers are not directly comparable.