SD12.4 · LESSON 4 OF 6

A file-processing pipeline

Produce an end-to-end design whose data paths, capacities and failure behavior satisfy explicit requirements.

THE IDEA

A file-processing pipeline

Send large content through an object-store path and keep metadata (owner, status, object key) in a separate record. Processing runs as a queued job with a durable identity, bounded retries and visible status transitions: uploaded → queued → processing → done, or failed and back to queued for a retry. Because the original is stored durably, a transient worker failure is retried against the same object: the client does not send the 2 GB again.

Step 1 of 7

1. client uploads the 2 GB original to the object store (stored durably under a key)

bytes the client sent: 2,000,000,000 large content goes through the object path; metadata and status live separately; every transition is visible and only legal ones are allowed (uploaded → queued → processing → done | failed → queued)

Step through a retry, with and without a re-upload.

Predict first. Should a transient worker failure force the client to upload the entire original again?

Chapter context

A case study integrates choices into one coherent system. Start with a small requirement set, trace the important operations, calculate the dominant costs and stress the design with a failure before adding components.

TRY IT YOURSELF

Part 1. Do it hands-on

Count bytes and attempts by hand, then write a job state machine with retries.

Step 1 of 7

1. client uploads the 2 GB original to the object store (stored durably under a key)

extra bytes if the whole file is re-sent extra bytes when the stored upload is reused attempts when the 2nd succeeds force a full re-upload on a transient failure? (yes/no)

To do: A 2 GB file is uploaded, then a worker fails once while processing it. If the client must upload the whole file again, extra bytes sent (decimal)?
To do: If the upload is stored durably in the object store and the retried job just refers to it, extra bytes the client sends?
To do: The job may be attempted 3 times in total. The first attempt fails (transient), the second succeeds. Attempts used?
To do: Should a transient worker failure force the client to upload the entire original again? Type yes or no

Part 2. Do it in code

TRANSITIONS (PROVIDED): uploaded → queued → processing → (done | failed); failed → queued (a retry); done is final.
class Job(job_id, object_key, size): state starts "uploaded"; move(to): ValueError for a transition not in TRANSITIONS; history is the list of states visited.
run(job, worker, max_attempts): queue the job, then up to max_attempts times: move to processing, call worker(job, attempt_number) (True = success). Success → done (return "done"). Failure → failed, and if attempts remain, retry through queued. After the last failure leave it failed and return "failed". The job REFERENCES the stored object (object_key); the upload is never repeated.
bytes_sent(size, worker_failures, reupload_on_failure): total bytes the client uploads: size, plus size per failure when the whole file is re-sent.
attempts_used(job): the number of times the job entered processing.

TRANSITIONS = {
    "uploaded": ["queued"],
    "queued": ["processing"],
    "processing": ["done", "failed"],
    "failed": ["queued"],
    "done": [],
}

class Job:
    def __init__(self, job_id, object_key, size):
        self.job_id = job_id
        self.object_key = object_key
        self.size = size
        self.state = "uploaded"
        self.history = ["uploaded"]

    def move(self, to):
        pass

def run(job, worker, max_attempts):
    pass

def bytes_sent(size, worker_failures, reupload_on_failure):
    pass

def attempts_used(job):
    pass

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Should a transient worker failure force the client to upload the entire original again?

AYesBNo

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For first-completion XP, define a unique logical event key such as (user_id, lesson_id, reward_version). Under a fixed reward r, $XP=r\times$ count(distinct eligible event keys). The deduplication record and awarded effect must share an atomic durable boundary in a real implementation.

Worked example

One hundred learners complete the same lesson, and every completion request is retried once. There are 200 deliveries but only 100 distinct learner-lesson events. At 20 XP per first completion, the correct total is 2000 XP, not 4000. The Python example models the invariant without pretending to implement distributed persistence.

Reference code

The following example uses the Python standard library and does not call a remote service.

deliveries=[]
for user in range(100):
    deliveries.extend([(user,'lesson-1','reward-v1')]*2)
unique=set(deliveries)
assert len(deliveries) == 200
assert len(unique) == 100
assert 20*len(unique) == 2000
assert 20*len(deliveries) == 4000

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A keyspace count does not prove collision freedom. Delivery and reading are different product events. Deduplication must cover both durable event identity and the actual effect.