L8.4 · LESSON 4 OF 6

Top-p

Compute temperature and candidate filtering, and distinguish sampling from greedy and beam decoding.

THE IDEA

Top-p (nucleus): keep enough probability

Top-p sorts the tokens from most to least probable and keeps the smallest prefix whose cumulative probability reaches p. The token that crosses the threshold stays in. Then renormalise and sample. The number of survivors adapts: few when the model is sure, many when it is not.

The worked example: logits [2, 1, 0] give [0.665, 0.245, 0.090]. With p = 0.8 the first token alone (0.665) is below the threshold, so the second joins; the survivors renormalise to about [0.731, 0.269].

p0.8

token (sorted)	probability	cumulative	kept?	after renormalising
1	0.6	0.6	yes	0.667
2	0.3	0.9	yes	0.333
3	0.1	1	no	0

keep the smallest prefix whose cumulative mass reaches p = 0.8: 2 token(s) The token that crosses the threshold is kept; at least one token always remains. Unlike a fixed k, the number kept adapts to how peaked the distribution is.

Move p over the distribution [0.6, 0.3, 0.1].

Predict first. For probabilities [0.6, 0.3, 0.1] and p = 0.8, which candidates remain?

Chapter context

Decoding chooses tokens from the model's output distribution. It changes generated behavior without retraining the model. Sampling settings influence diversity and concentration, but they do not independently guarantee correctness or creativity.

TRY IT YOURSELF

Part 1. Do it hands-on

Apply top-p by hand and choose p to keep exactly one token. Then code the filter and combine it with top-k.

p0.9

token (sorted)	probability	cumulative	kept?	after renormalising
1	0.6	0.6	yes	0.667
2	0.3	0.9	yes	0.333
3	0.1	1	no	0

keep the smallest prefix whose cumulative mass reaches p = 0.9: 2 token(s) The token that crosses the threshold is kept; at least one token always remains. Unlike a fixed k, the number kept adapts to how peaked the distribution is.

tokens kept at p = 0.8

renormalised survivors:

The slider above is for the third task (keep exactly one token).

To do: Probabilities [0.6, 0.3, 0.1] and p = 0.8. Sorted high to low, the first token reaches 0.6 (< 0.8), so we add the second (0.9 ≥ 0.8) and stop. How many tokens remain?
To do: Renormalise the two survivors (3 decimals)
To do: Choose p so that exactly ONE token survives

Part 2. Do it in code

top_p_filter(probs, p): sort the probabilities from high to low (ties: lower index first), keep the smallest prefix whose cumulative mass reaches p (the token that crosses the threshold is included), zero the rest and renormalise. p must be in (0, 1] (ValueError).
nucleus_size(probs, p): how many tokens survive.
combine(probs, k, p): apply top-k first, then top-p on the result (renormalise after each).
kept_indices(probs, p): the indices that survive, in descending probability order.

import numpy as np

def top_p_filter(probs, p):
    pass

def nucleus_size(probs, p):
    pass

def combine(probs, k, p):
    pass

def kept_indices(probs, p):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

For probabilities [.6,.3,.1] and p=.8, which candidates remain?

AAll three, because the threshold is below 1.BOnly the first, because it is the most probable.CThe first two: 0.6 is below 0.8, and adding 0.3 gives 0.9, which reaches the threshold.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

p_i(T)=\frac{\exp(z_i/T)}{\sum_j\exp(z_j/T)},\quad T>0.

Top-p retains a probability-sorted prefix with cumulative mass at least p, then renormalizes the retained entries. Specify tie handling and any minimum-candidate rule in implementation.

Worked example

For logits [2,1,0] at temperature one, probabilities are approximately [0.665241,0.244728,0.090031]. With top-p=0.8, keep the first two because the first alone is below the threshold. Their renormalized probabilities are approximately [0.731059,0.268941].

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import exp,isclose
weights=[exp(v-2) for v in [2,1,0]]
probs=[w/sum(weights) for w in weights]
assert probs[0] < .8 < sum(probs[:2])
retained=[p/sum(probs[:2]) for p in probs[:2]]
assert isclose(probs[0],.6652409557748218)
assert isclose(retained[0],.7310585786300049)
assert isclose(sum(retained),1)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Top-p=.8 does not mean keeping 80% of vocabulary items. A high temperature does not create new factual knowledge. Filter-and-renormalize order must be explicit.