L8.2 · LESSON 2 OF 6

Temperature

Compute temperature and candidate filtering, and distinguish sampling from greedy and beam decoding.

THE IDEA

Temperature

Before the softmax, the logits are divided by a temperature T > 0. T = 1 leaves the probabilities as the model produced them. T below 1 widens the gaps between logits, so the distribution becomes sharper and sampling is more predictable. T above 1 shrinks the gaps, so rarer tokens get a larger share. The ranking of the tokens never changes.

temperature T1logit 12logit 21logit 30

token	logit / T	probability
1	2	0.665
2	1	0.245
3	0	0.09

probabilities = softmax(logits / T); entropy = 0.832 nats low T → sharper (T → 0 approaches greedy; T = 0 itself is a division by zero) high T → flatter, the token ranking never changes

Logits [2, 1, 0] at different temperatures. Move the logits too.

Predict first. Can the formula directly divide by temperature zero?

Chapter context

Decoding chooses tokens from the model's output distribution. It changes generated behavior without retraining the model. Sampling settings influence diversity and concentration, but they do not independently guarantee correctness or creativity.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute probabilities at three temperatures by hand, then write the temperature softmax and measure sample frequencies.

temperature T1logit 12logit 21logit 30

token	logit / T	probability
1	2	0.665
2	1	0.245
3	0	0.09

probabilities = softmax(logits / T); entropy = 0.832 nats low T → sharper (T → 0 approaches greedy; T = 0 itself is a division by zero) high T → flatter, the token ranking never changes

softmax of [2, 1, 0]:

first probability at T = 0.5 first probability at T = 2

To do: Logits [2, 1, 0] at temperature 1. Type the three probabilities (3 decimals)
To do: Temperature 0.5 divides the logits first: [4, 2, 0]. Probability of the first token (3 decimals)
To do: Temperature 2 gives [1, 0.5, 0]. Probability of the first token (3 decimals)

Part 2. Do it in code

softmax(z): stable softmax of a vector.
apply_temperature(logits, T): softmax(logits / T). T must be positive (ValueError).
entropy(p): −Σ p ln p (0·ln 0 counts as 0). Higher entropy means a flatter distribution.
sample_with_temperature(logits, T, draws, seed): draw `draws` token indices from apply_temperature using np.random.default_rng(seed).choice and return the observed frequencies as an array.

import numpy as np

def softmax(z):
    pass

def apply_temperature(logits, T):
    pass

def entropy(p):
    pass

def sample_with_temperature(logits, T, draws, seed):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can the formula directly divide by temperature zero?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

p_i(T)=\frac{\exp(z_i/T)}{\sum_j\exp(z_j/T)},\quad T>0.

Top-p retains a probability-sorted prefix with cumulative mass at least p, then renormalizes the retained entries. Specify tie handling and any minimum-candidate rule in implementation.

Worked example

For logits [2,1,0] at temperature one, probabilities are approximately [0.665241,0.244728,0.090031]. With top-p=0.8, keep the first two because the first alone is below the threshold. Their renormalized probabilities are approximately [0.731059,0.268941].

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import exp,isclose
weights=[exp(v-2) for v in [2,1,0]]
probs=[w/sum(weights) for w in weights]
assert probs[0] < .8 < sum(probs[:2])
retained=[p/sum(probs[:2]) for p in probs[:2]]
assert isclose(probs[0],.6652409557748218)
assert isclose(retained[0],.7310585786300049)
assert isclose(sum(retained),1)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Top-p=.8 does not mean keeping 80% of vocabulary items. A high temperature does not create new factual knowledge. Filter-and-renormalize order must be explicit.