Let work wait without losing its identity
Picture
The API accepts a job and a worker processes it later. Arrival rate and processing rate no longer need to match at every instant.
Predict
Does acknowledging only after work automatically prevent duplicate effects?
Walk through the small example
No, the acknowledgment can be lost. The effect can happen before a redelivery is triggered.
Mental model and runnable Python model
Distinguish a work queue, where workers share jobs, from pub/sub, where each subscription receives its own stream. Define ordering scope, retention, acknowledgment, visibility timeout or lease, retry budget, dead-letter handling, and replay. At-least-once delivery requires duplicate-safe processing. Exactly-once claims have a scope and protocol; a queue cannot make arbitrary external side effects occur exactly once by slogan. Partitioning often provides per-key rather than global order. Backpressure should protect downstream systems instead of allowing an unbounded backlog.
What the function promises: The model demonstrates duplicate suppression in a single-threaded in-memory consumer. Its set and effects are not durable and do not constitute a crash-safe protocol.
class Consumer:
def __init__(self):
self.seen = set()
self.effects = []
def handle(self, job_id, payload):
if job_id in self.seen:
return False
self.effects.append(payload)
self.seen.add(job_id)
return True
consumer = Consumer()
assert consumer.handle("j7", "send reminder")
assert not consumer.handle("j7", "send reminder")
assert consumer.effects == ["send reminder"]
Work and storage: Expected O(1) dedup lookup and O(processed job IDs) retained state. A production retry window must align with dedup retention.
When to recognize it: A duplicate-safe consumer recognizes the same logical job. In a real system, recording completion and applying its effect need a shared durable boundary.
Check before applying it: A queue does not eliminate duplicates, overload, or partial external effects.
Change one rule
The process crashes after the effect but before seen.add. Is this toy consumer crash-safe?
See what changes
No. Commit a durable dedup record with an internal database effect, or use an idempotent external operation keyed by job identity. Some external effects still require reconciliation.
Connect
Share traffic across replaceable workers: Workers can scale separately from the rate at which requests arrive.
Design a notification pipeline: Notification delivery makes retries and acknowledgment uncertainty concrete.