cconnected.
Understand the pattern. Find your own way.Find your starting point
Your learning feed/Idea 207 of 218
Ideas this builds on
High-level design207 / 218

Make the system’s promises measurable

Picture

A successful API call is not enough if it arrives too late. Track request rate, errors, latency distributions, and resource saturation.

Predict

Can an average latency show how bad the slowest one percent feels?

Walk through the small example

Not reliably. Use an appropriate latency distribution or percentile for the tail.

Mental model and runnable Python model

Choose user-visible SLIs and an explicit SLO window. Track latency histograms, success/error categories, traffic, and saturation. Use structured logs with stable correlation IDs while redacting credentials and personal data. Distributed traces show critical-path spans but sampling can hide rare cases. Avoid user IDs or unbounded URLs as metric labels. Error-budget burn alerts should relate failures to the promised objective rather than paging on every isolated CPU spike. Telemetry must be designed for cost, retention, and access control.

What the function promises: The model computes a nearest-rank sample percentile and a request-based error budget. It is not a telemetry collector or a time-based availability calculation.

from math import ceil

def percentile(samples, p):
    if not samples or not 0 < p <= 100:
        raise ValueError("Need samples and 0 < percentile <= 100")
    values = sorted(samples)
    return values[ceil(p / 100 * len(values)) - 1]

def allowed_failures(requests, success_target):
    if requests < 0 or not 0 <= success_target <= 1:
        raise ValueError("Invalid objective")
    return requests * (1 - success_target)

assert percentile([10] * 9 + [1000], 99) == 1000
assert abs(allowed_failures(10000, 0.999) - 10) < 1e-8

Work and storage: O(n log n) sample sorting here. Production systems aggregate histograms or sketches rather than retaining every sample in one array.

When to recognize it: Metrics show trends. Logs record selected events. Traces connect spans across a request. Correlate them without leaking secrets or exploding label cardinality.

Check before applying it: Observability needs bounded cardinality, privacy, useful objectives, and actionable alerts—not just more logs.

Change one rule

A dashboard labels each metric with a unique request ID. What can go wrong?

See what changes

Unbounded series cardinality can overwhelm the metrics system and increase cost. Put per-request identity in controlled logs or traces, not every metric label.

Connect

Start with the system’s promises: Turn the stated user promise into a measurable signal and target.

Estimate the bottleneck before adding servers: Measurements validate or disprove the original capacity estimate.