cconnected.
Understand the pattern. Find your own way.Find your starting point
Your learning feed/Idea 202 of 218
Ideas this builds on
High-level design202 / 218

Contain failure instead of multiplying it

Picture

Requests keep waiting, new requests pile up, and the caller becomes unhealthy too. Give downstream work a bounded deadline.

Predict

Should every layer independently retry an operation three times without a shared budget?

Walk through the small example

No, retries can multiply. Nested retries can create a large amplification of one request.

Mental model and runnable Python model

Use timeouts within an end-to-end deadline, bounded retry budgets, exponential backoff with jitter, and idempotency for uncertain writes. Circuit breakers reduce repeated pressure on a failing dependency; bulkheads limit which resources a failure can consume. Distinguish availability from durability and define RTO (time to recover) and RPO (acceptable recovery-point loss). Replicas are not backups, and backups without restore tests are incomplete recovery planning. The sample circuit is sequential: a distributed or concurrent breaker needs careful probe coordination and measured thresholds.

What the function promises: Circuit.call executes a supplied local operation with caller-provided monotonic time. After threshold failures it opens for cooldown seconds. It models one sequential breaker, not retries or a network stack.

class Circuit:
    def __init__(self, threshold=2, cooldown=5):
        if threshold < 1 or cooldown <= 0:
            raise ValueError("Invalid breaker settings")
        self.threshold, self.cooldown = threshold, cooldown
        self.failures, self.open_until = 0, 0
    def call(self, now, operation):
        if now < self.open_until:
            raise RuntimeError("Circuit open")
        try:
            result = operation()
        except Exception:
            self.failures += 1
            if self.failures >= self.threshold:
                self.open_until = now + self.cooldown
            raise
        self.failures, self.open_until = 0, 0
        return result

circuit = Circuit()
def fail():
    raise ValueError("Dependency failed")
for _ in range(2):
    try:
        circuit.call(0, fail)
    except ValueError:
        pass
assert circuit.open_until == 5
assert circuit.call(5, lambda: "recovered") == "recovered"
assert circuit.failures == 0

Work and storage: O(1) local breaker bookkeeping; the supplied operation has its own cost.

When to recognize it: After enough failures, reject quickly for a cooldown. Then permit a controlled probe before trusting the dependency again.

Check before applying it: Timeouts, retries, circuit breakers, and isolation need coordinated budgets, not independent defaults.

Change one rule

Many threads reach the cooldown boundary simultaneously. Does this sequential model guarantee one probe?

See what changes

No. Coordinate half-open probe admission atomically and bound concurrency. Otherwise a recovery probe can become another traffic burst.

Connect

Let work wait without losing its identity: Backlog and redelivery can amplify a dependency failure.

Give traffic a refillable budget: Admission control limits the traffic entering an overloaded system.