A6.4 · LESSON 4 OF 6

Human review

Identify instruction-injection attempts and design permission, sandbox, approval and evaluation boundaries for an agent.

THE IDEA

Human review

When the task's authorisation does not settle whether an action is acceptable, put a person at the effect boundary: purchases above a limit, irreversible actions, anything outside the original scope. Show the concrete proposed change and its consequences ("refund $20 to order A17"), not a vague goal. Then make sure what runs is what was approved: a different amount or target needs a fresh approval.

review threshold ($)50

$20: auto$50: needs review$120: needs review$5: auto

2 of 4 need review, 2 are approved automatically

amount the agent actually executes ($)20

reviewer approved: "refund $20 to order A17" agent executes: "refund $20 to order A17" same fingerprint: the approval covers it

Require review at meaningful effect boundaries, and show the concrete change and its consequences. Approving a vague goal is not approving every side effect.

Move the threshold, then change the executed amount.

Predict first. Is approving a vague goal identical to approving every possible side effect?

Chapter context

An agent combines untrusted content with tools that can affect real systems. Reliable operation requires boundaries enforced outside model text, alongside clear task authorization and evidence-based evaluation.

TRY IT YOURSELF

Part 1. Do it hands-on

Triage actions by hand, then write the review policy and an approval check.

review threshold ($)50

$20: auto$50: needs review$120: needs review$5: auto

2 of 4 need review, 2 are approved automatically

amount the agent actually executes ($)20

reviewer approved: "refund $20 to order A17" agent executes: "refund $20 to order A17" same fingerprint: the approval covers it

Require review at meaningful effect boundaries, and show the concrete change and its consequences. Approving a vague goal is not approving every side effect.

need review ($20, $50, $120, $5) approved automatically $20 approved, $200 executed: valid? (yes/no) a vague goal approves all effects? (yes/no)

To do: Policy: purchases of $50 or more need human review. Purchases of $20, $50, $120 and $5 are proposed. How many need review?
To do: How many are approved automatically?
To do: The reviewer approved "refund $20 to order A17". The agent then executes "refund $200 to order A17". Is the approval valid for what ran? Type yes or no
To do: Is approving a vague goal ("handle my refunds") identical to approving every possible side effect? Type yes or no

Part 2. Do it in code

needs_review(action, threshold=50): True for an irreversible action (action['irreversible'] true) or an amount at or above the threshold.
triage(actions): returns {"review": [indices needing review], "auto": [indices that do not]}.
render_approval(action): a concrete, human-readable request: "<verb> $<amount> to <target>" using action['verb'], action['amount'] (shown with no decimals if whole) and action['target'].
fingerprint(action): a stable string of verb, amount and target joined with "|".
approval_valid(approved, executed): True only if the executed action has the same fingerprint as the approved one (a changed amount or target invalidates the approval).

def needs_review(action, threshold=50):
    pass

def triage(actions):
    pass

def render_approval(action):
    pass

def fingerprint(action):
    pass

def approval_valid(approved, executed):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Is approving a vague goal identical to approving every possible side effect?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a stated evaluation set:

unsafe\ pass\ rate=\frac{unsafe\ cases\ that\ pass}{unsafe\ cases},\quad false\ block\ rate=\frac{benign\ cases\ blocked}{benign\ cases}.

These empirical rates characterize that set; they are not universal security guarantees. Define denominators and outcome criteria before running the evaluation.

Worked example

An offline evaluation contains 100 unauthorized-action cases, of which 2 pass, and 200 benign cases, of which 5 are blocked. Unsafe pass rate is 2%, and false block rate is 2.5%. Those two numbers measure different failures; averaging them into a single undocumented 'safety score' hides the tradeoff.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
unsafe_passes,unsafe_cases=2,100
benign_blocks,benign_cases=5,200
assert isclose(unsafe_passes/unsafe_cases,.02)
assert isclose(benign_blocks/benign_cases,.025)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A successful prompt-injection defense on one dataset is not a proof for every input. Text instructions cannot substitute for permission enforcement. Audit logs must preserve evidence without exposing credentials.