cconnected.
Understand the pattern. Find your own way.Find your starting point
Your learning feed/Idea 200 of 218
Ideas this builds on
High-level design200 / 218

Let work wait without losing its identity

Picture

The API accepts a job and a worker processes it later. Arrival rate and processing rate no longer need to match at every instant.

Predict

Does acknowledging only after work automatically prevent duplicate effects?

Walk through the small example

No, the acknowledgment can be lost. The effect can happen before a redelivery is triggered.

Mental model and runnable Python model

Distinguish a work queue, where workers share jobs, from pub/sub, where each subscription receives its own stream. Define ordering scope, retention, acknowledgment, visibility timeout or lease, retry budget, dead-letter handling, and replay. At-least-once delivery requires duplicate-safe processing. Exactly-once claims have a scope and protocol; a queue cannot make arbitrary external side effects occur exactly once by slogan. Partitioning often provides per-key rather than global order. Backpressure should protect downstream systems instead of allowing an unbounded backlog.

What the function promises: The model demonstrates duplicate suppression in a single-threaded in-memory consumer. Its set and effects are not durable and do not constitute a crash-safe protocol.

class Consumer:
    def __init__(self):
        self.seen = set()
        self.effects = []
    def handle(self, job_id, payload):
        if job_id in self.seen:
            return False
        self.effects.append(payload)
        self.seen.add(job_id)
        return True

consumer = Consumer()
assert consumer.handle("j7", "send reminder")
assert not consumer.handle("j7", "send reminder")
assert consumer.effects == ["send reminder"]

Work and storage: Expected O(1) dedup lookup and O(processed job IDs) retained state. A production retry window must align with dedup retention.

When to recognize it: A duplicate-safe consumer recognizes the same logical job. In a real system, recording completion and applying its effect need a shared durable boundary.

Check before applying it: A queue does not eliminate duplicates, overload, or partial external effects.

Change one rule

The process crashes after the effect but before seen.add. Is this toy consumer crash-safe?

See what changes

No. Commit a durable dedup record with an internal database effect, or use an idempotent external operation keyed by job identity. Some external effects still require reconciliation.

Connect

Share traffic across replaceable workers: Workers can scale separately from the rate at which requests arrive.

Design a notification pipeline: Notification delivery makes retries and acknowledgment uncertainty concrete.