SD8.1 · LESSON 1 OF 6

Queues and logs

Design asynchronous processing with explicit acknowledgment, ordering, replay and duplicate-handling rules.

THE IDEA

Queues and logs

Asynchronous work needs somewhere to wait. A work queue hands each job to one worker and forgets it once it is acknowledged. A retained log appends messages and keeps them; each consumer tracks its own position, so a new consumer can read from the start and an old one can replay. Real systems often mix both patterns, so check what yours actually guarantees.

The same five messages, handled as a work queue and as a retained log.

work queue: jobs acknowledged3log: consumer A's position3

work queue: job 1job 2job 3job 4job 5

log: offset 0offset 1offset 2offset 3offset 4

work queue: 2 job(s) left; acknowledged jobs are removed and cannot be replayed log: A is at offset 3, 2 unread; a new consumer B starts at 0 and can read all 5; replay = move a position back a log keeps messages when they are read; a work queue forgets them when they are acknowledged

Move the acknowledgements and the consumer position.

Predict first. Does acknowledging a work-queue message necessarily preserve it forever for replay?

Chapter context

Asynchronous work decouples when a request is accepted from when its side effects finish. Queues and logs buffer differences in timing, but they do not remove processing cost or guarantee globally unique effects by themselves.

TRY IT YOURSELF

Part 1. Do it hands-on

Count what is left and what can be replayed by hand, then build a work queue and a log.

The same five messages, handled as a work queue and as a retained log.

work queue: jobs acknowledged3log: consumer A's position3

work queue: job 1job 2job 3job 4job 5

log: offset 0offset 1offset 2offset 3offset 4

unread by A at position 3 messages a new consumer can replay jobs left after 3 acks of 5 an ack preserves it for replay? (yes/no)

To do: A retained log holds 5 messages (offsets 0–4). Consumer A has processed up to offset 2 and its position is 3. Messages still unread by A?
To do: A brand-new consumer B starts at the beginning of the same log. Messages it can read (replay)?
To do: A work queue holds 5 jobs and 3 are acknowledged (done). Jobs left in the queue?
To do: Does acknowledging a work-queue message necessarily preserve it forever for replay? Type yes or no

Part 2. Do it in code

class WorkQueue: push(job), pop() → the next job (it becomes in-flight; None if empty), ack(job) removes it for good, requeue(job) puts an unacknowledged in-flight job back at the FRONT, pending() → jobs waiting, in_flight() → jobs delivered but not acknowledged.
class Log: append(msg) → its offset; read(consumer, n) → up to n messages from that consumer's position, advancing it (new consumers start at 0); position(consumer); unread(consumer); replay(consumer) → reset the position to 0 and return the full history.
A log never removes messages when a consumer reads them.

class WorkQueue:
    def __init__(self):
        self.items = []
        self.flight = []

    def push(self, job):
        pass

    def pop(self):
        pass

    def ack(self, job):
        pass

    def requeue(self, job):
        pass

    def pending(self):
        pass

    def in_flight(self):
        pass

class Log:
    def __init__(self):
        self.messages = []
        self.pos = {}

    def append(self, msg):
        pass

    def read(self, consumer, n):
        pass

    def position(self, consumer):
        pass

    def unread(self, consumer):
        pass

    def replay(self, consumer):
        pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does acknowledging a work-queue message necessarily preserve it forever for replay?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A fluid approximation for backlog B is

\Delta B\approx(\lambda-\mu)\Delta t

while the queue is nonempty, where lambda is arrival rate and mu processing rate. Backlog cannot go below zero. Delivery guarantees must specify which system boundary and side effects they cover.

Worked example

Arrivals at 120 jobs/s and processing at 100 jobs/s grow the queue by 20 jobs/s, reaching 1200 after one minute. If arrivals then drop to 80 jobs/s, net drain rate is 20 jobs/s and clearing that backlog takes 60 seconds under the constant-rate model.

Reference code

The following example uses the Python standard library and does not call a remote service.

arrival,service,seconds=120,100,60
backlog=max(0,(arrival-service)*seconds)
assert backlog == 1200
new_arrival=80
drain=service-new_arrival
assert drain == 20 and backlog/drain == 60

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Exactly-once terminology must state its scope. Retrying can duplicate external effects. A partition's order is not a universal order across an entire system.