SD8.2 · LESSON 2 OF 6

Acknowledgments

Design asynchronous processing with explicit acknowledgment, ordering, replay and duplicate-handling rules.

THE IDEA

Acknowledgments

When does the consumer tell the broker "done"? Acknowledge first and a crash after the ack but before processing loses the message for good (at-most-once). Process first and a crash before the ack makes the broker redeliver; if the work had already happened, it happens twice (at-least-once). Most systems choose at-least-once and handle the duplicates in the consumer.

Step 1 of 4

1. receive m2

processed: m1, m2, m2, m3 lost: none m2 processed twice at-least-once: never loses work, may repeat it, so the consumer must tolerate duplicates

Pick the order and when the crash happens.

Predict first. Can one accepted message be delivered more than once under common at-least-once contracts?

Chapter context

Asynchronous work decouples when a request is accepted from when its side effects finish. Queues and logs buffer differences in timing, but they do not remove processing cost or guarantee globally unique effects by themselves.

TRY IT YOURSELF

Part 1. Do it hands-on

Trace crashes by hand, then simulate both acknowledgement orders.

Step 1 of 4

1. receive m2

processed: m1, m2, m2, m3 lost: none m2 processed twice at-least-once: never loses work, may repeat it, so the consumer must tolerate duplicates

messages lost (ack first, crash on m2) messages processed (ack last, crash before finishing) times m2 processed (crash after finishing) can a message be delivered twice? (yes/no)

To do: Messages m1, m2, m3. The consumer ACKNOWLEDGES first and then processes. It crashes right after acknowledging m2 and before processing it. How many messages are lost for good?
To do: Now it processes first and acknowledges afterwards, and crashes while handling m2 BEFORE finishing it. After the restart the broker redelivers m2. Messages processed in the end?
To do: Same order (process, then acknowledge) but it crashes AFTER finishing m2 and before acknowledging. Times m2 is processed in total?
To do: Can one accepted message be delivered more than once under a common at-least-once contract? Type yes or no

Part 2. Do it in code

run(msgs, mode, crash_msg=None, crash_when="before_process"): simulate one consumer. mode "ack_first": acknowledge then process; mode "ack_last": process then acknowledge. The consumer crashes ONCE while handling crash_msg and then restarts.
ack_first: if it crashes on crash_msg, that message was already acknowledged, so it is never redelivered and never processed (lost).
ack_last: a crash with crash_when="before_process" processes nothing for that message and the broker redelivers it; with crash_when="after_process" the message was processed but not acknowledged, so it is redelivered and processed again.
Return {"processed": the list of processing events in order (duplicates included), "lost": messages never processed, "duplicates": messages processed more than once (sorted)}.
delivery_guarantee(mode): "at-most-once" for ack_first, "at-least-once" for ack_last.

def run(msgs, mode, crash_msg=None, crash_when="before_process"):
    pass

def delivery_guarantee(mode):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can one accepted message be delivered more than once?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A fluid approximation for backlog B is

\Delta B\approx(\lambda-\mu)\Delta t

while the queue is nonempty, where lambda is arrival rate and mu processing rate. Backlog cannot go below zero. Delivery guarantees must specify which system boundary and side effects they cover.

Worked example

Arrivals at 120 jobs/s and processing at 100 jobs/s grow the queue by 20 jobs/s, reaching 1200 after one minute. If arrivals then drop to 80 jobs/s, net drain rate is 20 jobs/s and clearing that backlog takes 60 seconds under the constant-rate model.

Reference code

The following example uses the Python standard library and does not call a remote service.

arrival,service,seconds=120,100,60
backlog=max(0,(arrival-service)*seconds)
assert backlog == 1200
new_arrival=80
drain=service-new_arrival
assert drain == 20 and backlog/drain == 60

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Exactly-once terminology must state its scope. Retrying can duplicate external effects. A partition's order is not a universal order across an entire system.