A6.6 · LESSON 6 OF 6

Observability and incidents

Identify instruction-injection attempts and design permission, sandbox, approval and evaluation boundaries for an agent.

THE IDEA

Observability and incidents

When something goes wrong you need a record: the task id, the actions that were allowed, the decisions made and pointers to evidence of the effects, without secret values. With that you can roll back what is reversible (delete the row, then the file), compensate where you cannot (refund the charge) and know what cannot be undone (the email). And you investigate the cause rather than only silencing the error message.

effects that happened before the failure4

happened, in order: file → email → charge → row rollback: delete row then file; compensate charge; cannot be undone: email

trace: { task_id: "t-17", action: "call_api", decision: "allow", api_key: "***", evidence: ["row:884"] }

Keep task ids, allowed actions, decisions and pointers to effect evidence, without secret values. Plan rollback or compensation where operations allow it, and investigate why a failure happened instead of only suppressing its message.

Choose how far the run got, and toggle redaction.

Predict first. Should secret values be copied into traces for convenience?

Chapter context

An agent combines untrusted content with tools that can affect real systems. Reliable operation requires boundaries enforced outside model text, alongside clear task authorization and evidence-based evaluation.

TRY IT YOURSELF

Part 1. Do it hands-on

Plan a rollback by hand, then write redaction, event logging and the rollback plan.

effects that happened before the failure4

happened, in order: file → email → charge → row rollback: delete row then file; compensate charge; cannot be undone: email

trace: { task_id: "t-17", action: "call_api", decision: "allow", api_key: "***", evidence: ["row:884"] }

effects deletable effects that cannot be undone first undone (file or row) copy secrets into traces? (yes/no)

To do: Effects in order: (1) create file, (2) send email, (3) charge card, (4) write database row. Reversible by deleting: the file and the row. The card can be refunded (compensation); the email cannot be unsent. How many effects can be rolled back by deleting?
To do: How many cannot be undone at all?
To do: Undoing the two deletable effects in reverse order, which is undone first? Type file or row
To do: Should secret values (API keys) be copied into traces for convenience? Type yes or no

Part 2. Do it in code

redact(record, secret_keys): a NEW dict where every key whose lower-cased name contains one of the secret key fragments (e.g. "key", "token", "password") has its value replaced by "***". Nested dicts are redacted too. The input is not modified.
log_event(task_id, action, decision, evidence=None): {"task_id", "action", "decision", "evidence"} — a record with the task id, what was attempted, the decision and pointers to effect evidence (an empty list if none).
rollback_plan(effects): effects is the list of {"name", "undo"} in the order they happened; undo is "delete", "compensate" or "none". Return {"delete": names to delete in REVERSE order, "compensate": names needing a compensating action (reverse order), "irreversible": names that cannot be undone (original order)}.
incident_summary(events): events are log_event records; return {"total": count, "blocked": count with decision "block", "tasks": sorted unique task ids}.

def redact(record, secret_keys):
    pass

def log_event(task_id, action, decision, evidence=None):
    pass

def rollback_plan(effects):
    pass

def incident_summary(events):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Should secret values be copied into traces for convenience?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a stated evaluation set:

unsafe\ pass\ rate=\frac{unsafe\ cases\ that\ pass}{unsafe\ cases},\quad false\ block\ rate=\frac{benign\ cases\ blocked}{benign\ cases}.

These empirical rates characterize that set; they are not universal security guarantees. Define denominators and outcome criteria before running the evaluation.

Worked example

An offline evaluation contains 100 unauthorized-action cases, of which 2 pass, and 200 benign cases, of which 5 are blocked. Unsafe pass rate is 2%, and false block rate is 2.5%. Those two numbers measure different failures; averaging them into a single undocumented 'safety score' hides the tradeoff.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
unsafe_passes,unsafe_cases=2,100
benign_blocks,benign_cases=5,200
assert isclose(unsafe_passes/unsafe_cases,.02)
assert isclose(benign_blocks/benign_cases,.025)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A successful prompt-injection defense on one dataset is not a proof for every input. Text instructions cannot substitute for permission enforcement. Audit logs must preserve evidence without exposing credentials.