Errors and observations
Trace a model-requested tool call through schema validation, execution and a returned observation.
THE IDEA
Errors and observations
What the tool returns is the model's only view of what happened, an observation. When something fails, return it in a defined, flagged form so the next step can recover or stop honestly. Tell failures apart: a domain error (the tool ran and said no), a transport failure (it could not be reached, so a retry may help) and invalid arguments (fix the call). Hiding a failure, or answering with an invented success, lets later steps build on a fiction.
Distinguish a tool's own refusal (domain), an unreachable service (transport) and a malformed request (invalid). The model needs the error in a defined, flagged form to recover or stop honestly.
Predict first. Should a failed lookup be silently returned as an invented successful result?
Chapter context
A tool call is a structured request for the application to perform an operation. The model proposes arguments; the surrounding application decides whether the operation is valid and permitted and is responsible for executing it.
TRY IT YOURSELF
Live lab
Atlantis has no weather data, so the tool fails. What the model does next depends on whether your host tells it about the failure.
TRY IT YOURSELF
Report failures honestly
Atlantis has no weather data, so the tool fails. What the model does next depends on whether your host tells it about the failure.
Get a final answer that says the lookup failed instead of inventing a temperature. This is graded by a simple text rule: no made-up numbers, and an admission of failure.
Task given to the model: What's the weather in Atlantis right now?
This loop as code
import anthropic
client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY
TOOLS = [ # name, description and input_schema are the "Tool definition"
{"name": t["name"], "description": t["description"], "input_schema": t["schema"]}
for t in [
{
"name": "get_weather",
"description": "Get the current weather for a city.",
"schema": {
"type": "object",
"properties": {
"city": {
"type": "string",
"description": "City name"
},
"unit": {
"type": "string",
"enum": [
"celsius",
"fahrenheit"
]
}
},
"required": [
"city"
]
}
}
]
]
messages = [{"role": "user", "content": "What's the weather in Atlantis right now?"}]
for step in range(4): # Step budget
response = client.messages.create(model="claude-haiku-4-5", max_tokens=600, tools=TOOLS, messages=messages)
messages.append({"role": "assistant", "content": response.content})
calls = [b for b in response.content if b.type == "tool_use"]
if not calls:
break # final answer
results = []
for call in calls: # the model only PROPOSES these
problems = validate(call.input, schema_for(call.name)) # your host-side check
if problems:
output, is_error = "Invalid arguments: " + "; ".join(problems), True
else:
output, is_error = run_tool(call.name, call.input) # your code runs it
results.append({"type": "tool_result", "tool_use_id": call.id, "content": output, "is_error": is_error})
messages.append({"role": "user", "content": results})Exercises
Part 1. Do it hands-on
Classify three failures, then write the executor that reports each kind of failure honestly.
Distinguish a tool's own refusal (domain), an unreachable service (transport) and a malformed request (invalid). The model needs the error in a defined, flagged form to recover or stop honestly.
- To do: Classify each failure as domain (the tool ran and said no), transport (could not reach the service) or invalid (bad arguments). The weather lookup for "Atlantis" returns "no data for Atlantis"
- To do: The call to the weather service times out
- To do: The model sends {"town": "Oslo"} and the host rejects it before running anything
- To do: Should a failed lookup be silently returned as an invented successful result? Type yes or no
Part 2. Do it in code
- DomainError and TransportError (PROVIDED) are exceptions a tool can raise: the tool ran and refused, or the service could not be reached.
- execute_tool(tools, schemas, name, args): return {"content": str, "is_error": bool, "kind": str}. Unknown tool: kind "unknown", content "unknown tool: <name>". Arguments that fail validate_args (PROVIDED): kind "invalid", content "invalid arguments: " + ", ".join(problems). A DomainError: kind "domain", content str(e). A TransportError: kind "transport", content str(e). Any other exception: kind "internal", content "internal error". Success: kind "ok", content str(result), is_error False. Every other kind has is_error True.
- model_view(result, report_errors): what the model is shown. With report_errors True return {"content": result["content"], "is_error": result["is_error"]}. With False an error is swallowed: return {"content": "", "is_error": False}; a success is returned unchanged.
- retryable(kind): True only for "transport" (the same call may succeed later).
- answer_is_honest(answer, failed): if failed is True the answer must admit it (contain one of: could not, couldn't, unable, failed, no data, not available — case-insensitive) AND contain no digit. If failed is False return True.
Both the live lab and the exercises must be solved to finish this lesson.
Stuck, or something not working? You can come back to it any time.
CHECK YOUR UNDERSTANDING
Should a failed lookup be silently returned as an invented successful result?
Chapter reference: math, worked example & code
This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.
Math and notation
A useful execution model is: requested operation + validated arguments + permission decision → execution result. Schema validation establishes a data-shape contract; authorization establishes whether this caller may perform this operation. Neither implies the other.
Worked example
A calculator tool requires numeric a and b and returns a+b. For a=2,b=3, its result is 5. If the request instead asks for an unknown field or omits b, reject or repair the request according to the contract before executing. A retry of this pure calculation is harmless; that does not establish the same property for a side-effecting tool.
Reference code
The following example uses the Python standard library and does not call a remote service.
def add(a,b):
if type(a) not in (int,float) or type(b) not in (int,float):
raise ValueError('numeric inputs required')
return a+b
assert add(2,3) == 5
try:
add('2',3)
except ValueError:
pass
else:
raise AssertionError('invalid input accepted')
Python examples are provided for study and local execution; this page does not execute Python.
Common mistakes
A model proposal is not permission. JSON schema shape validation is not business-rule validation. Tool outputs must not silently become trusted instructions.