L8.6 · LESSON 6 OF 6

Constraints and repetition

Compute temperature and candidate filtering, and distinguish sampling from greedy and beam decoding.

THE IDEA

Repetition penalties and constrained decoding

Models can loop, repeating a phrase. A repetition penalty lowers the logit of tokens that have already appeared (dividing positive logits, multiplying negative ones, so both become less likely). Constrained or structured decoding goes further: at each step only tokens the grammar allows may be chosen, by setting every other logit to −∞, which gives them probability exactly 0.

repetition penalty1.5

token	already used?	logit	after penalty	probability
the	yes	3	2	0.5
cat	no	2	2	0.5
sat	no	1	1	0
mat	yes	-1	-1.5	0

penalty: positive logits are divided, negative ones multiplied, so seen tokens always become less likely masking sets disallowed logits to −∞ → probability exactly 0

{"capital": "Lyon"}valid JSONnot correct

Structured decoding guarantees the shape of the output, not that its content is true.

Adjust the penalty, and mask tokens to see them disappear from the distribution.

Predict first. Does grammatical validity prove semantic correctness?

Chapter context

Decoding chooses tokens from the model's output distribution. It changes generated behavior without retraining the model. Sampling settings influence diversity and concentration, but they do not independently guarantee correctness or creativity.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute a penalised logit and count masked tokens, then implement repetition penalties, logit masking and a JSON validity check.

repetition penalty1.5

token	already used?	logit	after penalty	probability
the	yes	3	2	0.5
cat	no	2	2	0.5
sat	no	1	1	0
mat	yes	-1	-1.5	0

penalty: positive logits are divided, negative ones multiplied, so seen tokens always become less likely masking sets disallowed logits to −∞ → probability exactly 0

{"capital": "Lyon"}valid JSONnot correct

Structured decoding guarantees the shape of the output, not that its content is true.

3.0 with penalty 1.5 masked tokens (50,000 vocabulary, 11 allowed) valid JSON? correct?

To do: A repetition penalty divides a positive logit of a token that already appeared by 1.5. A logit of 3.0 becomes…
To do: Structured decoding: after the characters {"a": the grammar allows only a digit (10 symbols 0–9) or a quote. Of a 50,000-token vocabulary where only these 11 symbols are allowed, how many are masked out?
To do: A constrained decoder produces {"capital": "Lyon"} for "capital of France". Is it valid JSON (yes/no), and is it correct (yes/no)? Type two words, e.g. "yes no"

Part 2. Do it in code

repetition_penalty(logits, seen, penalty): for every token id in seen: if its logit is positive divide it by penalty, otherwise multiply it by penalty. penalty must be at least 1 (ValueError). Return a new array.
mask_logits(logits, allowed): set the logit of every token not in `allowed` to −inf. ValueError if allowed is empty.
constrained_pick(logits, allowed): the allowed token id with the highest logit (first on ties).
is_valid_json(text): True if json.loads accepts the text, else False. (This checks format only.)

import json
import numpy as np

def repetition_penalty(logits, seen, penalty):
    pass

def mask_logits(logits, allowed):
    pass

def constrained_pick(logits, allowed):
    pass

def is_valid_json(text):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does grammatical validity prove semantic correctness?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

p_i(T)=\frac{\exp(z_i/T)}{\sum_j\exp(z_j/T)},\quad T>0.

Top-p retains a probability-sorted prefix with cumulative mass at least p, then renormalizes the retained entries. Specify tie handling and any minimum-candidate rule in implementation.

Worked example

For logits [2,1,0] at temperature one, probabilities are approximately [0.665241,0.244728,0.090031]. With top-p=0.8, keep the first two because the first alone is below the threshold. Their renormalized probabilities are approximately [0.731059,0.268941].

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import exp,isclose
weights=[exp(v-2) for v in [2,1,0]]
probs=[w/sum(weights) for w in weights]
assert probs[0] < .8 < sum(probs[:2])
retained=[p/sum(probs[:2]) for p in probs[:2]]
assert isclose(probs[0],.6652409557748218)
assert isclose(retained[0],.7310585786300049)
assert isclose(sum(retained),1)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Top-p=.8 does not mean keeping 80% of vocabulary items. A high temperature does not create new factual knowledge. Filter-and-renormalize order must be explicit.