A2.4 · LESSON 4 OF 6

Memory and context

Design a bounded observe-act loop with explicit state, stopping conditions and a verifiable task outcome.

THE IDEA

Memory and context

The text currently inside the model's context window is not the same as the agent's persistent task state. A window holds a limited number of tokens, so old messages fall out. With messages of 40, 30, 50 and 20 tokens and a window of 100, the first message (40 tokens) is silently lost, along with whatever it said. Save identifiers and results somewhere durable (a small memory), pin the essential ones into every prompt, and write a fact down before its message is dropped.

context window (tokens)100

0: order id A17 (40)1: ship to Lyon (30)2: chit-chat (50)3: gift wrap (20)

window 100 − 10 pinned = 90 tokens for messages → keeps messages [2, 3] (70 tokens), drops [0, 1] the order id fell out of the window, but it survives in the pinned memory

The text inside the model's context is not a database. Store identifiers and results in persistent task state, and re-supply them, so truncation or summarisation cannot silently erase task facts.

Shrink the window and pin a fact.

Predict first. Does a long chat automatically provide a reliable database?

Chapter context

An agent chooses actions based on a task and observations, often over several model calls. Useful autonomy requires bounded resources, explicit feedback and a stopping rule that recognizes both success and inability to continue.

TRY IT YOURSELF

Part 1. Do it hands-on

Work out which message falls out of a window, then build a context that pins facts and saves them to memory.

context window (tokens)100

0: order id A17 (40)1: ship to Lyon (30)2: chit-chat (50)3: gift wrap (20)

window 100 − 10 pinned = 90 tokens for messages → keeps messages [2, 3] (70 tokens), drops [0, 1] the order id fell out of the window, but it survives in the pinned memory

The text inside the model's context is not a database. Store identifiers and results in persistent task state, and re-supply them, so truncation or summarisation cannot silently erase task facts.

messages kept at 100 tokens index of the dropped message tokens lost a long chat is a reliable database? (yes/no)

To do: A conversation has four messages of 40, 30, 50 and 20 tokens (oldest first). The context window holds 100 tokens and keeps the NEWEST messages that fit. How many messages are kept?
To do: Which message (index from 0, oldest first) falls out of the window?
To do: How many tokens of conversation are silently lost?
To do: Does a long chat automatically provide a reliable database of the facts in it? Type yes or no

Part 2. Do it in code

fit_window(sizes, budget): the indices (ascending) of the NEWEST messages that fit in the token budget (stop at the first one that does not fit).
dropped(sizes, budget): the indices that fell out of the window.
build_context(pinned_sizes, msg_sizes, budget): pinned facts are always included and use budget first; the rest of the budget holds the newest messages. Return {"pinned": all pinned indices, "messages": kept message indices}. ValueError if the pinned items alone exceed the budget.
evict(messages, sizes, budget, memory): messages are dicts {"text", "facts": {...}}. Messages that fall out of the window have their facts copied into the memory dict BEFORE they are dropped (later facts overwrite earlier ones for the same key). Return the list of kept messages.
recall(memory, key, default=None): the stored value or the default.

def fit_window(sizes, budget):
    pass

def dropped(sizes, budget):
    pass

def build_context(pinned_sizes, msg_sizes, budget):
    pass

def evict(messages, sizes, budget, memory):
    pass

def recall(memory, key, default=None):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does a long chat automatically provide a reliable database?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

Represent a run as states s0,s1,... and actions a0,a1,... with observed transitions. A budget constraint can bound the sum of measured call costs. History-dependent token growth means a constant per-step estimate is only a simplifying model.

Worked example

A simulated agent may take at most three actions. It moves from room A to B, inspects a clue, then moves to C. The trace contains three actions. If the objective still is not satisfied, the correct result is budget exhausted with partial progress, not a fourth unapproved action or a fabricated success.

Reference code

The following example uses the Python standard library and does not call a remote service.

actions=['move:A->B','inspect:clue','move:B->C']
budget=3
assert len(actions) == budget
objective_satisfied=False
status='complete' if objective_satisfied else 'budget-exhausted'
assert status == 'budget-exhausted'
assert len(actions)*.002 == .006

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A plan is not execution evidence. Reflection can repeat the original mistake. A step budget is useful only if every executed action is counted consistently.