Estimate the bottleneck before adding servers
Picture
At 100 requests per second and 0.2 seconds average service time, about 20 requests are active in a stable system.
Predict
Does average concurrency alone guarantee a p99 latency target?
Walk through the small example
No. Queueing, skew, pauses, and bursts affect the tail.
Mental model and runnable Python model
Estimate peak reads/writes per second, mean service time, in-flight work, storage growth, bandwidth, and hot-key concentration. Little’s Law L=λW applies to stable averages with consistent system boundaries. For workers handling one request at a time, a first estimate is peak_rate×mean_service_time/target_utilization, rounded up; async multiplexing and resource-specific bottlenecks require a different model. Separate CPU, database connections, I/O, memory, and network limits. Add explicit failure headroom and verify with representative load tests rather than copying internet capacity numbers.
What the function promises: The model estimates single-request worker count and raw replicated payload storage using stated assumptions. It excludes indexes, compression, metadata, and operational overhead.
from math import ceil
def estimate_capacity(peak_rps, mean_seconds, utilization,
writes_per_day, bytes_per_write, days, replicas):
if (peak_rps < 0 or mean_seconds < 0 or not 0 < utilization < 1
or min(writes_per_day, bytes_per_write, days) < 0 or replicas < 1):
raise ValueError("Invalid assumptions")
return {
"workers": ceil(peak_rps * mean_seconds / utilization),
"raw_bytes": writes_per_day * bytes_per_write * days * replicas,
}
result = estimate_capacity(100, 0.2, 0.5, 1000, 200, 30, 3)
assert result == {"workers": 40, "raw_bytes": 18000000}
Work and storage: Constant-sized arithmetic, followed by real measurement work outside the model.
When to recognize it: Estimate bytes per write, retention, replication, indexes, and overhead. Also estimate read bandwidth and the hottest key, not only total requests.
Check before applying it: Capacity arithmetic is a hypothesis with units and assumptions, not a substitute for load tests.
Change one rule
The workload becomes heavily skewed toward one partition. Is the total average utilization still enough?
See what changes
No. Measure the hottest shard, key, tenant, and operation. A lightly loaded cluster can still contain one saturated partition.
Connect
Share traffic across replaceable workers: Worker throughput depends on service time and available headroom.
Design a social feed: Fan-out and hot authors can dominate a social feed’s actual workload.