cconnected.
Understand the pattern. Find your own way.Find your starting point
Your learning feed/Idea 217 of 218
Ideas this builds on
High-level design217 / 218

Design a streaming counter

Picture

An event created at second 12 may arrive after one created at second 18. Arrival order and event time are different.

Predict

Does processing arrival order automatically preserve the time when events actually occurred?

Walk through the small example

No. Network and queue delays can reorder events.

Mental model and runnable Python model

Define event identity, timestamp semantics, aggregation keys, windows, permitted lateness, and correction policy. Partition by aggregate key to keep updates coordinated, while recognizing hot keys may need partial aggregation and a merge step. Deduplication state and offsets/checkpoints must recover consistently with output effects. Watermarks are protocol assumptions, not a universal guarantee that no older event will ever arrive. Keep provisional and finalized results distinguishable. Global exactly-once effects depend on the full source-state-sink protocol and replay behavior.

Contract and event schema

Define event_id, aggregate key, event timestamp, producer version, and payload. Choose processing-time or event-time windows, allowed lateness, and whether late data is dropped, corrected, or sent for offline recomputation.

Processing flow

Ingest into a durable partitioned log, validate and deduplicate, assign windows, update keyed state, and publish provisional or final aggregates. Partition ordering is not global ordering, and hot keys may require partial aggregation.

Recovery

Coordinate checkpointed offsets, dedup state, aggregate state, and sink updates. Replaying input must not duplicate committed effects. A transactional sink or idempotent versioned aggregate write may support the required guarantee.

Serving and operations

Readers need freshness and finality metadata, not only a number. Monitor lag, late-event rate, state size, hot partitions, and correction backlog. Retention must cover the promised replay and correction windows.

What the function promises: WindowCounter is a local event-time model with fixed-size windows, stable event IDs, and a max-observed-time minus lateness watermark. It retains all state for demonstration and drops too-late events.

class WindowCounter:
    def __init__(self, width, lateness):
        if width <= 0 or lateness < 0:
            raise ValueError("Invalid window")
        self.width, self.lateness = width, lateness
        self.maximum = 0
        self.seen, self.counts = set(), {}
    def add(self, event_id, event_time):
        if event_time < 0:
            raise ValueError("Invalid event time")
        if event_id in self.seen or event_time < self.maximum - self.lateness:
            return False
        self.maximum = max(self.maximum, event_time)
        self.seen.add(event_id)
        window = event_time // self.width * self.width
        self.counts[window] = self.counts.get(window, 0) + 1
        return True

counter = WindowCounter(10, 5)
assert counter.add("a", 12)
assert counter.add("b", 18)
assert not counter.add("a", 12)
assert not counter.add("late", 9)
assert counter.counts == {10: 2}

Work and storage: Expected O(1) local event updates. Production state must be retired according to finalized windows, replay, and dedup retention rules.

When to recognize it: A watermark is an explicit progress assumption. Events older than the accepted lateness boundary may be dropped or handled through a correction path.

Check before applying it: Window finality, late-event policy, dedup retention, and checkpoint recovery must agree.

Change one rule

We delete dedup IDs before the source’s replay window ends. What can happen?

See what changes

A replayed old event can be counted again unless finalized-window policy prevents it. Coordinate retention across source offsets, aggregate state, and dedup records.

Connect

Let work wait without losing its identity: Replay and duplicate delivery change aggregate correctness.

Ask which version a read is allowed to see: A fresh-looking count may still be provisional or eventually consistent.