Designing Ad Click Aggregation: Exactly-Once Counting at Scale

Counting sounds like the easiest problem in distributed systems until every count is money. Ad click aggregation — bill advertisers per click, in near-real-time, at ~10K-1M events/sec — is the canonical "streaming with correctness" interview. The twist that makes it hard: retries, replays, and crashes all try to count the same click twice, and an advertiser double-billed at Black Friday scale is a lawsuit, not a bug.

Requirements

Functional: count clicks per (ad_id, minute); query top-N ads per window; support late events and dispute investigations. Non-functional: end-to-end freshness of ~1 minute, no lost clicks, no double counting (the billing pipeline's correctness bar), events can arrive minutes late or out of order.

Pipeline shape

click -> collector API -> Kafka (raw log, keyed by ad_id)
      -> stream aggregator (Flink) -> OLAP store (billing + dashboards)
      -> raw events also land in object storage (the audit trail)

Kafka is the spine for the usual reason: it decouples a spiky write path from a stateful consumer and makes replay possible. The design interview is really about three correctness problems layered on this pipe.

Problem 1: the same click, twice

Duplicates enter at every hop: the browser retries the beacon, the collector retries the Kafka produce, the aggregator reprocesses after a crash, a consumer replays a partition. Deduplication needs an event ID minted at the edge (click ID generated in the ad SDK), then two lines of defense:

• Idempotent produce (Kafka's producer idempotence) kills the collector-retry duplicates.
• Keyed dedupe state in the aggregator: keep click IDs seen in the last N minutes (per ad partition) and drop repeats. State is bounded by the dedupe window — you cannot remember forever, so you pick a window (say 15 minutes) that covers realistic retry horizons and accept that a duplicate arriving 3 hours late is caught by reconciliation instead.

A Bloom filter per window is the classic space fix if exact state is too large — with the explicit caveat that false positives now drop a real click at ~0.1%, which is a business decision, not an engineering one. For billing, exact state wins.

Problem 2: exactly-once between systems

Inside Flink, "exactly-once" is real: periodic checkpoints snapshot operator state and Kafka offsets atomically; on crash, both rewind together, so every event affects state exactly once. The hole is the boundary — state inside Flink and rows in your OLAP store are two different systems.

Two workable seams:

1. Transactional sink: Flink's two-phase-commit sink ties "write results" and "advance checkpoint" together (Kafka-to-Kafka does this natively).
2. Idempotent sink (simpler, usually better): make the write an upsert keyed by (ad_id, window_start). Reprocessing then overwrites the same row with the same value — duplicates become harmless. Exactly-once effects via at-least-once delivery + idempotent writes; in an interview, saying it that way shows you know the theorem and the workaround.

Problem 3: time

Event time (when the click happened) ≠ processing time (when it arrived). Aggregate by event time with watermarks: "I have probably seen everything up to T." Windows close when the watermark passes; events later than that are late.

window [12:00, 12:01) closes at watermark 12:01+allowed_lateness
late click at 12:07 for the 12:00 window -> side output -> correction job

Late events go to a side channel that emits corrections (upsert the window row again), not silent drops. Billing tolerates a row being revised at 12:07; it does not tolerate the revision never happening.

The reconciliation layer

Even with all of the above, mature systems run a batch pass over the raw log (the object-storage copy) that recomputes each hour/day and diffs against the streaming results — lambda architecture surviving in its most defensible niche. Streaming answers "what do dashboards and pacing decisions see now"; batch answers "what do we actually invoice." Discrepancy above a threshold pages someone. This is also your dispute story: the raw immutable log is the evidence, aggregates are derived and re-derivable.

Query side

(ad_id, minute) aggregates land in an OLAP store built for this shape — pre-aggregated rows with time-partitioned rollups (minute → hour → day) to keep top-N and range scans cheap. Druid/Pinot/ClickHouse are the usual suspects; the design point is that you query aggregates, never raw events, on the hot path.

Interview probe	The answer they want
Why not count in Redis INCR?	No replay, no event-time windows, INCR+retry = double count
Hot ad (one key, 100K clicks/sec)?	Pre-aggregate per task before keyed shuffle; split key by subkey then roll up
Exactly-once, really?	At-least-once + idempotent upserts + dedupe window + batch reconciliation
Where's the source of truth?	Immutable raw log; everything else is a rebuildable view

That last row is the theme worth closing on: streaming outputs are caches of a computation over an immutable log. Get the log right, make every downstream write idempotent, and "exactly-once" stops being marketing and starts being a property you can defend line by line.