A6.5 · LESSON 5 OF 6

Evaluations

Identify instruction-injection attempts and design permission, sandbox, approval and evaluation boundaries for an agent.

THE IDEA

Evaluations

Test the agent on ordinary tasks, attacks, malformed inputs and attempts at unauthorised actions, and measure two failures separately: unsafe actions that got through and benign work that was wrongly blocked. With 2 of 100 attacks passing and 5 of 200 benign tasks blocked, the rates are 2% and 2.5%. A policy that blocks everything scores a perfect 0% unsafe and a useless 100% false blocks, which is why one blended "safety score" hides the trade-off.

attacks that got through (of 100)2benign tasks wrongly blocked (of 200)5

agent	unsafe pass rate	false block rate
allow everything	100%	0%
block everything	0%	100%
your policy	2%	2.5%

blocking everything scores a perfect 0% on unsafe passes and a useless 100% on false blocks averaging the two into one "safety score" would give block-all and allow-all the same 50%: report both numbers

Include ordinary tasks, attacks, malformed inputs and unauthorised-action attempts, and measure both kinds of failure.

Move the two counts and compare with the trivial policies.

Predict first. Does blocking every request produce a useful agent?

Chapter context

An agent combines untrusted content with tools that can affect real systems. Reliable operation requires boundaries enforced outside model text, alongside clear task authorization and evidence-based evaluation.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute both rates by hand, then write the evaluator and a Pareto comparison.

attacks that got through (of 100)2benign tasks wrongly blocked (of 200)5

agent	unsafe pass rate	false block rate
allow everything	100%	0%
block everything	0%	100%
your policy	2%	2.5%

Include ordinary tasks, attacks, malformed inputs and unauthorised-action attempts, and measure both kinds of failure.

unsafe pass rate (2 of 100) false block rate (5 of 200) false block rate of block-everything block-everything is useful? (yes/no)

To do: 100 unauthorised-action attempts, 2 got through. Unsafe pass rate?
To do: 200 benign tasks, 5 were wrongly blocked. False block rate?
To do: An agent that blocks EVERY request: what is its false block rate on the 200 benign tasks?
To do: Does blocking every request produce a useful agent? Type yes or no

Part 2. Do it in code

rates(cases): cases is a list of {"attack": bool, "allowed": bool} (allowed = the agent let it through). Return {"unsafe_pass_rate": allowed attacks / attacks, "false_block_rate": blocked benign / benign}; a zero denominator gives 0.
block_everything(cases) and allow_everything(cases): the rates of the two trivial baselines on the same cases.
meets_slo(r, max_unsafe, max_false_block): both rates within their limits.
pareto(rows): rows is a list of {"name", "unsafe", "false_block"}; return the names not dominated by another row (another row is no worse on both and strictly better on at least one), in input order.
describe(r): "unsafe pass X%, false block Y%" with one decimal place each, reporting both numbers separately.

def rates(cases):
    pass

def block_everything(cases):
    pass

def allow_everything(cases):
    pass

def meets_slo(r, max_unsafe, max_false_block):
    pass

def pareto(rows):
    pass

def describe(r):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does blocking every request produce a useful agent?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a stated evaluation set:

unsafe\ pass\ rate=\frac{unsafe\ cases\ that\ pass}{unsafe\ cases},\quad false\ block\ rate=\frac{benign\ cases\ blocked}{benign\ cases}.

These empirical rates characterize that set; they are not universal security guarantees. Define denominators and outcome criteria before running the evaluation.

Worked example

An offline evaluation contains 100 unauthorized-action cases, of which 2 pass, and 200 benign cases, of which 5 are blocked. Unsafe pass rate is 2%, and false block rate is 2.5%. Those two numbers measure different failures; averaging them into a single undocumented 'safety score' hides the tradeoff.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
unsafe_passes,unsafe_cases=2,100
benign_blocks,benign_cases=5,200
assert isclose(unsafe_passes/unsafe_cases,.02)
assert isclose(benign_blocks/benign_cases,.025)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A successful prompt-injection defense on one dataset is not a proof for every input. Text instructions cannot substitute for permission enforcement. Audit logs must preserve evidence without exposing credentials.