Contain failure instead of multiplying it
Picture
Requests keep waiting, new requests pile up, and the caller becomes unhealthy too. Give downstream work a bounded deadline.
Predict
Should every layer independently retry an operation three times without a shared budget?
Walk through the small example
No, retries can multiply. Nested retries can create a large amplification of one request.
Mental model and runnable Python model
Use timeouts within an end-to-end deadline, bounded retry budgets, exponential backoff with jitter, and idempotency for uncertain writes. Circuit breakers reduce repeated pressure on a failing dependency; bulkheads limit which resources a failure can consume. Distinguish availability from durability and define RTO (time to recover) and RPO (acceptable recovery-point loss). Replicas are not backups, and backups without restore tests are incomplete recovery planning. The sample circuit is sequential: a distributed or concurrent breaker needs careful probe coordination and measured thresholds.
What the function promises: Circuit.call executes a supplied local operation with caller-provided monotonic time. After threshold failures it opens for cooldown seconds. It models one sequential breaker, not retries or a network stack.
class Circuit:
def __init__(self, threshold=2, cooldown=5):
if threshold < 1 or cooldown <= 0:
raise ValueError("Invalid breaker settings")
self.threshold, self.cooldown = threshold, cooldown
self.failures, self.open_until = 0, 0
def call(self, now, operation):
if now < self.open_until:
raise RuntimeError("Circuit open")
try:
result = operation()
except Exception:
self.failures += 1
if self.failures >= self.threshold:
self.open_until = now + self.cooldown
raise
self.failures, self.open_until = 0, 0
return result
circuit = Circuit()
def fail():
raise ValueError("Dependency failed")
for _ in range(2):
try:
circuit.call(0, fail)
except ValueError:
pass
assert circuit.open_until == 5
assert circuit.call(5, lambda: "recovered") == "recovered"
assert circuit.failures == 0
Work and storage: O(1) local breaker bookkeeping; the supplied operation has its own cost.
When to recognize it: After enough failures, reject quickly for a cooldown. Then permit a controlled probe before trusting the dependency again.
Check before applying it: Timeouts, retries, circuit breakers, and isolation need coordinated budgets, not independent defaults.
Change one rule
Many threads reach the cooldown boundary simultaneously. Does this sequential model guarantee one probe?
See what changes
No. Coordinate half-open probe admission atomically and bound concurrency. Otherwise a recovery probe can become another traffic burst.
Connect
Let work wait without losing its identity: Backlog and redelivery can amplify a dependency failure.
Give traffic a refillable budget: Admission control limits the traffic entering an overloaded system.