A5.6 · LESSON 6 OF 6

Evaluation

Choose when to delegate independent work and define handoffs, shared-state rules and evaluation.

THE IDEA

Evaluation

Judge a multi-agent system on final task success, integration quality, cost and latency. Parallelism may cut latency (11 minutes to 6) without raising reliability: three independent 90%-reliable steps in a row all succeed only 0.729 of the time. It also costs more: $0.07 for three workers and an orchestrator against $0.03 for one agent. And agreement between agents proves little when they share a model or a source.

per-step reliability0.9steps / agents in a chain3

all 3 steps must succeed: 0.729 at least one of 3 independent attempts succeeds: 0.999 (only if their failures are independent) cost: 3 workers × $0.02 + orchestrator $0.01 = $0.07 against $0.03 for one agent (2.33×)

Parallelism can cut latency without improving reliability, and it costs more. Workers that share a model, a prompt or a source can agree on the same wrong answer, so agreement is not proof of correctness. Measure final task success, integration quality, cost and latency.

Chain steps and compare the maths.

Predict first. Can several agents confidently repeat the same wrong source?

Chapter context

Delegation can divide a task into focused work units, but coordination creates costs and new failure paths. Add helpers when their independent contributions can be integrated and checked more clearly than one agent can do the work alone.

TRY IT YOURSELF

Part 1. Do it hands-on

Work out reliability and cost by hand, then write the evaluation helpers, including an agreement-versus-correctness report.

per-step reliability0.9steps / agents in a chain3

three 90% steps all succeed multi-agent cost ($) times the single-agent cost agents can repeat the same wrong source? (yes/no)

To do: Three independent steps are each 90% reliable and all must succeed. Probability all succeed (3 decimals)?
To do: A multi-agent run: three workers at $0.02 each plus an orchestrator at $0.01. Total cost in dollars?
To do: A single agent doing the same job costs $0.03. The multi-agent run costs how many times as much (2 decimals)?
To do: Can several agents confidently repeat the same wrong source? Type yes or no

Part 2. Do it in code

chain_success(ps): the probability that independent steps with success probabilities ps all succeed (product).
any_success(ps): the probability that at least one of several independent attempts succeeds (1 − ∏(1 − p)).
run_cost(worker_costs, orchestrator_cost): the total cost.
cost_ratio(multi, single): multi / single.
agreement_report(runs): runs is a list of {"answers": [...], "truth": value}. Return {"all_agree": fraction of runs where every answer is identical, "majority_correct": fraction where the most common answer (ties: first seen) equals the truth, "confidently_wrong": fraction where all answers agree AND differ from the truth}.

def chain_success(ps):
    pass

def any_success(ps):
    pass

def run_cost(worker_costs, orchestrator_cost):
    pass

def cost_ratio(multi, single):
    pass

def agreement_report(runs):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can several agents confidently repeat the same wrong source?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

With independent parallel task durations $t_1,\ldots,t_n$ and merge time m, an ideal lower-bound schedule is $\max_i t_i+m$ , excluding orchestration overhead. If independent tasks each succeed with probability p, all n succeed with probability $p^n$ ; correlated failures invalidate that simple model.

Worked example

Three tasks take 2,5 and 3 minutes; integration takes 1. Ideal sequential duration is 11 minutes, while fully independent parallel duration is 6. Separately, three independent 90%-reliable components all succeed with probability 0.9^3=0.729. The latency gain does not itself establish a reliability gain.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose
times,merge=[2,5,3],1
assert sum(times)+merge == 11
assert max(times)+merge == 6
assert isclose(.9**3,.729)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Parallel workers need explicit ownership of shared artifacts. Agreement can reflect correlated failure. More agents can increase token usage and integration work even when wall-clock time falls.