SD8.5 · LESSON 5 OF 6

Outbox and sagas

Design asynchronous processing with explicit acknowledgment, ordering, replay and duplicate-handling rules.

THE IDEA

Outbox and sagas

Saving a row and then publishing an event are two separate steps: a crash between them leaves the database and the message system disagreeing. The outbox pattern records the event in the same transaction as the data and publishes it afterwards, at-least-once. A multi-step workflow such as reserve, charge, ship can be a saga: if a step fails, the steps already done are undone by compensating actions, in reverse order. A compensation is a new action, not a perfect rewind.

1. INSERT order o1 (committed) 2. CRASH 3. publish order_created: never happens the row exists, the event does not: two systems disagree

saga steps reserve → charge → ship, failing at:

completed: reserve compensations (reverse order): undo reserve the compensating action is another step, not a perfect rewind (a refund may keep a fee, an email stays sent)

Crash between the row and the event; fail a saga step.

Predict first. Does writing a row then separately publishing an event form an atomic pair by default?

Chapter context

Asynchronous work decouples when a request is accepted from when its side effects finish. Queues and logs buffer differences in timing, but they do not remove processing cost or guarantee globally unique effects by themselves.

TRY IT YOURSELF

Part 1. Do it hands-on

Reason about crashes and compensations by hand, then write the naive pair, the outbox, the relay and a saga.

1. INSERT order o1 (committed) 2. CRASH 3. publish order_created: never happens the row exists, the event does not: two systems disagree

saga steps reserve → charge → ship, failing at:

completed: reserve compensations (reverse order): undo reserve the compensating action is another step, not a perfect rewind (a refund may keep a fee, an email stays sent)

after the crash: rows/events (like 1/0) compensations if the charge fails compensations if shipping fails row + publish atomic by default? (yes/no)

To do: Code writes an order row, then separately publishes an "order created" event. It crashes between the two steps. Rows saved / events published afterwards? Type like 1/0
To do: A saga runs reserve-stock, charge-card, ship. The card charge FAILS. Which completed steps need a compensating action? How many?
To do: Instead, shipping fails (after reserve and charge). Compensating actions needed?
To do: Does writing a row and then separately publishing an event form an atomic pair by default? Type yes or no

Part 2. Do it in code

naive_publish(db, bus, order, crash_between): write the order into db['orders'] and then append an event to bus; if crash_between is true raise RuntimeError between the two steps. (Two separate steps, no atomicity.)
outbox_write(db, order, crash_before_commit=False): in ONE atomic step append the order to db['orders'] and an {"event": ..., "sent": False} record to db['outbox']; if crash_before_commit is true raise RuntimeError and leave db completely unchanged.
relay(db, bus, crash_after_publish=False): publish every outbox record that is not marked sent, then mark it sent; if crash_after_publish is true raise RuntimeError after the first publish and before marking it (so a later relay publishes it again: at-least-once). Return the number published in this call.
run_saga(steps, fail_at): steps is a list of step names; fail_at the index of the step that fails (or None). Return {"done": steps completed before the failure, "compensated": the compensating actions run for the completed steps in REVERSE order}.

import copy

def naive_publish(db, bus, order, crash_between=False):
    pass

def outbox_write(db, order, crash_before_commit=False):
    pass

def relay(db, bus, crash_after_publish=False):
    pass

def run_saga(steps, fail_at):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does writing a row then separately publishing an event form an atomic pair by default?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A fluid approximation for backlog B is

\Delta B\approx(\lambda-\mu)\Delta t

while the queue is nonempty, where lambda is arrival rate and mu processing rate. Backlog cannot go below zero. Delivery guarantees must specify which system boundary and side effects they cover.

Worked example

Arrivals at 120 jobs/s and processing at 100 jobs/s grow the queue by 20 jobs/s, reaching 1200 after one minute. If arrivals then drop to 80 jobs/s, net drain rate is 20 jobs/s and clearing that backlog takes 60 seconds under the constant-rate model.

Reference code

The following example uses the Python standard library and does not call a remote service.

arrival,service,seconds=120,100,60
backlog=max(0,(arrival-service)*seconds)
assert backlog == 1200
new_arrival=80
drain=service-new_arrival
assert drain == 20 and backlog/drain == 60

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Exactly-once terminology must state its scope. Retrying can duplicate external effects. A partition's order is not a universal order across an entire system.