A4.6 · LESSON 6 OF 6

Skill evaluation

Write a scoped skill package and manage its instructions and references within a task's context budget.

THE IDEA

Skill evaluation

A skill can fail in three ways: not activating when it should, activating when it should not, or activating and getting the task wrong. Build a small test set of relevant and unrelated tasks and measure activation recall and the false-activation rate, then check the observable outcome of each task. Review any bundled script before it gets execution authority: shipping a script is not evidence that it works or that it is safe.

Click a task to toggle whether the skill activated on it. The first five tasks are ones where it should; the last five are unrelated.

hits 4 misses 1 false activations 1 correct rejections 4 recall 0.8 false-activation rate 0.2 accuracy 0.8

Test that the skill activates on appropriate tasks, stays quiet on unrelated ones and produces correct observable outcomes. A script bundled in the skill has not been run or reviewed just because it is there.

Toggle activations and watch the metrics.

Predict first. Does including a script in a skill mean it has already run successfully?

Chapter context

A skill packages reusable task instructions and optional supporting material. Context engineering decides which instructions and evidence are available for a particular operation. Loading more text is not automatically better than selecting the right text.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute activation metrics by hand, then write the evaluator and a script reviewer.

Click a task to toggle whether the skill activated on it. The first five tasks are ones where it should; the last five are unrelated.

hits 4 misses 1 false activations 1 correct rejections 4 recall 0.8 false-activation rate 0.2 accuracy 0.8

recall (4 of 5) false-activation rate (1 of 5) accuracy a bundled script has already run? (yes/no)

To do: Ten test tasks: 5 where the skill should activate, 5 unrelated. It activates on 4 of the 5 relevant ones. Activation recall?
To do: It also activates on 1 of the 5 unrelated tasks. False-activation rate?
To do: Overall activation accuracy (correct decisions out of 10)?
To do: Does including a script in a skill mean it has already run successfully? Type yes or no

Part 2. Do it in code

evaluate_activation(cases): cases is a list of {"relevant": bool, "activated": bool}. Return {"tp", "fp", "fn", "tn", "recall" (tp/(tp+fn)), "false_rate" (fp/(fp+tn)), "accuracy"}; a ratio with a zero denominator is 0.
outcome_ok(result, expected): True if every key in expected has the same value in result (the observable outcome, not the skill's own claim).
review_script(source): the sorted list of risky patterns present in the script text from this list: "rm -rf", "curl ", "wget ", "eval(", "os.system", "subprocess", "chmod 777".
ready(report, min_recall, max_false_rate): True if recall >= min_recall and false_rate <= max_false_rate.

def evaluate_activation(cases):
    pass

def outcome_ok(result, expected):
    pass

def review_script(source):
    pass

def ready(report, min_recall, max_false_rate):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does including a script in a skill mean it has already run successfully?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For a synthetic combined budget W, a bookkeeping model is available = W − instructions − history − tool results − reserved output. Actual provider contracts can separate these limits, so use the chosen model's documented accounting when implementing.

Worked example

A synthetic 8192-token budget reserves 1200 for instructions, 2800 for history, 1600 for tool results and 1500 for output. Remaining space is 1092 tokens. A 900-token skill body fits under this model; adding a 2000-token reference does not. The remedy is to select or retrieve the relevant passage rather than silently dropping task constraints.

Reference code

The following example uses the Python standard library and does not call a remote service.

window=8192
used=1200+2800+1600+1500
remaining=window-used
assert remaining == 1092
assert 900 <= remaining
assert 2000 > remaining

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Compaction can lose evidence and constraints. Skill installation is not task authorization. A context limit and a product's memory feature are different mechanisms.