A6.1 · LESSON 1 OF 6

Prompt injection

Identify instruction-injection attempts and design permission, sandbox, approval and evaluation boundaries for an agent.

THE IDEA

Prompt injection

Prompt injection is text that tries to redirect the agent away from its authorised task. It can sit in retrieved pages, tool output, emails or any processed content. The model reads everything as text, so it cannot be relied on to tell "the user asked" from "the page says". The robust defence is a boundary in the application: the task decides which actions are allowed, and requests that came from data are blocked or confirmed rather than obeyed.

#	requested by text in a page	decision
1	send_email	block
2	send_email	block
3	delete_file	block
4	read_page	ask
5	send_email	block
6	transfer	block
7	delete_file	block
8	summarise	ask
9	send_email	block
10	transfer	block

attack success rate: 0/10 = 0 task allow-list: read_page, summarise (the page asked for send_email, delete_file, read_page, transfer, summarise) the text cannot grant itself authority: unlisted actions are blocked, listed ones asked about

Injected instructions can hide in retrieved text, tool output or any processed content. The defence is a boundary the text cannot move: authorised actions decided by the task, with extra confirmation when the request came from data.

Ten injected requests, with and without a boundary.

Predict first. Does an instruction inside a search result automatically outrank the user's task?

Chapter context

An agent combines untrusted content with tools that can affect real systems. Reliable operation requires boundaries enforced outside model text, alongside clear task authorization and evidence-based evaluation.

TRY IT YOURSELF

Part 1. Do it hands-on

Measure attack success with and without a policy by hand, then write the policy and its evaluation.

#	requested by text in a page	decision
1	send_email	block
2	send_email	block
3	delete_file	block
4	read_page	ask
5	send_email	block
6	transfer	block
7	delete_file	block
8	summarise	ask
9	send_email	block
10	transfer	block

success rate with no policy run without confirmation with the policy blocked outright a search-result instruction outranks the user? (yes/no)

To do: Ten injected requests arrive inside retrieved pages (each asks for an action). With no policy, every requested action runs. Attack success rate?
To do: With a policy that never lets data-origin text trigger an action directly (allowed actions: read_page, summarise; data-origin requests must be confirmed), how many of the 10 run without confirmation?
To do: How many of the 10 requests are blocked outright (the action is not in the task's allow-list)?
To do: Does an instruction inside a search result automatically outrank the user's task? Type yes or no

Part 2. Do it in code

decide(action, origin, allowed): origin is "user" or "data". An action not in the task's allow-list is "block" whatever its origin. An allowed action requested by the user is "allow"; an allowed action that was requested by data (a page, a tool result) is "ask" (needs confirmation).
evaluate(attacks, allowed): attacks is a list of {"action", "origin"}. Return {"ran": number decided allow, "asked": number ask, "blocked": number block, "success_rate": ran / total (0 for an empty list)}.
without_policy(attacks): the success rate when every requested action simply runs (1.0 for a non-empty list).
extract_requests(text, known_actions): the sorted known action names that appear in the text as whole words (these are only REQUESTS found in data; they carry no authority).

import re

def decide(action, origin, allowed):
    pass

def evaluate(attacks, allowed):
    pass

def without_policy(attacks):
    pass

def extract_requests(text, known_actions):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does an instruction inside a search result automatically outrank the user's task?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a stated evaluation set:

unsafe\ pass\ rate=\frac{unsafe\ cases\ that\ pass}{unsafe\ cases},\quad false\ block\ rate=\frac{benign\ cases\ blocked}{benign\ cases}.

These empirical rates characterize that set; they are not universal security guarantees. Define denominators and outcome criteria before running the evaluation.

Worked example

An offline evaluation contains 100 unauthorized-action cases, of which 2 pass, and 200 benign cases, of which 5 are blocked. Unsafe pass rate is 2%, and false block rate is 2.5%. Those two numbers measure different failures; averaging them into a single undocumented 'safety score' hides the tradeoff.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
unsafe_passes,unsafe_cases=2,100
benign_blocks,benign_cases=5,200
assert isclose(unsafe_passes/unsafe_cases,.02)
assert isclose(benign_blocks/benign_cases,.025)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A successful prompt-injection defense on one dataset is not a proof for every input. Text instructions cannot substitute for permission enforcement. Audit logs must preserve evidence without exposing credentials.