SD8.3 · LESSON 3 OF 6

Idempotent consumers

Design asynchronous processing with explicit acknowledgment, ordering, replay and duplicate-handling rules.

THE IDEA

Idempotent consumers

If a message can arrive twice, applying it twice must not matter. The usual tool is deduplication on an operation identity: remember which ids were applied and skip repeats. It only works if the record survives crashes and is written atomically with the effect. Payment p1 (+50) delivered twice gives 100 without a check, 50 with a durable atomic check, and 100 again if the processed-id set was only in memory or was recorded in a separate step that a crash skipped.

message p1 (+50) is delivered twice final balance: 50 correct: the effect happened once the id and the balance change together, so a repeat is always recognised

A message id is only useful if something stores it and checks it before every effect, and the check-and-record must survive crashes and be atomic with the effect.

Try the four consumers, with and without a restart.

Predict first. Does a message ID alone prevent duplicate effects without storage and checks?

Chapter context

Asynchronous work decouples when a request is accepted from when its side effects finish. Queues and logs buffer differences in timing, but they do not remove processing cost or guarantee globally unique effects by themselves.

TRY IT YOURSELF

Part 1. Do it hands-on

Compute balances for four designs by hand, then write them and watch the weak ones fail.

message p1 (+50) is delivered twice final balance: 50 correct: the effect happened once the id and the balance change together, so a repeat is always recognised

A message id is only useful if something stores it and checks it before every effect, and the check-and-record must survive crashes and be atomic with the effect.

balance, no dedup durable atomic dedup memory-only set + restart crash between balance and id an id alone prevents duplicates? (yes/no)

To do: Payment message p1 (+50) is delivered twice to a consumer that does no deduplication. Starting balance 0. Final balance?
To do: With a processed-ids check that is stored together with the balance in one atomic update?
To do: The processed-ids set lives only in memory and the consumer restarts between the two deliveries. Final balance?
To do: If the balance is updated first and the processed-id is saved in a SEPARATE later step, and the consumer crashes between them, then redelivery adds the 50 again. Final balance?
To do: Does a message ID alone prevent duplicate effects without storage and checks? Type yes or no

Part 2. Do it in code

Each consumer class has handle(msg_id, amount) that adds amount to self.balance and restart() that simulates a crash and restart.
NoDedup: always applies the amount.
MemoryDedup: remembers processed ids in a Python set; restart() clears that set (memory is lost) but the balance survives (it is stored durably).
DurableDedup: keeps self.store = {"balance": ..., "seen": [...]} and updates BOTH in one step (all or nothing); restart() keeps the store.
NonAtomicDedup: applies the amount to the balance first and records the id in a SEPARATE step afterwards; handle(msg_id, amount, crash_between=False) with crash_between True raises RuntimeError after the balance step and before the id is recorded.
All classes start with balance 0 and expose .balance (a property is fine).

class NoDedup:
    def __init__(self):
        self.balance = 0

    def handle(self, msg_id, amount):
        pass

    def restart(self):
        pass

class MemoryDedup:
    def __init__(self):
        self.balance = 0
        self.seen = set()

    def handle(self, msg_id, amount):
        pass

    def restart(self):
        pass

class DurableDedup:
    def __init__(self):
        self.store = {"balance": 0, "seen": []}

    @property
    def balance(self):
        return self.store["balance"]

    def handle(self, msg_id, amount):
        pass

    def restart(self):
        pass

class NonAtomicDedup:
    def __init__(self):
        self.balance = 0
        self.seen = []

    def handle(self, msg_id, amount, crash_between=False):
        pass

    def restart(self):
        pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does a message ID alone prevent duplicate effects without storage and checks?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A fluid approximation for backlog B is

\Delta B\approx(\lambda-\mu)\Delta t

while the queue is nonempty, where lambda is arrival rate and mu processing rate. Backlog cannot go below zero. Delivery guarantees must specify which system boundary and side effects they cover.

Worked example

Arrivals at 120 jobs/s and processing at 100 jobs/s grow the queue by 20 jobs/s, reaching 1200 after one minute. If arrivals then drop to 80 jobs/s, net drain rate is 20 jobs/s and clearing that backlog takes 60 seconds under the constant-rate model.

Reference code

The following example uses the Python standard library and does not call a remote service.

arrival,service,seconds=120,100,60
backlog=max(0,(arrival-service)*seconds)
assert backlog == 1200
new_arrival=80
drain=service-new_arrival
assert drain == 20 and backlog/drain == 60

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Exactly-once terminology must state its scope. Retrying can duplicate external effects. A partition's order is not a universal order across an entire system.