Design an Ad Click Aggregator

Commerce, Payments & LedgersHarddesign-ad-click-aggregator

Design click ingestion and near-real-time aggregation where the numbers are billed: deduplicated events, late arrivals, fraud filtering, and a batch layer that corrects the stream.

Stream AggregationLambda ArchitectureDeduplicationWatermarks

Traffic & Capacity Estimates:

1M clicks/second peak · dashboards within 1 minute · billing-grade accuracy

Functional Requirements

•Ingest click and impression events with advertiser, campaign, and context dimensions.
•Aggregate by campaign and time window for near-real-time dashboards.
•Deduplicate retried and replayed events, and handle events arriving minutes late.
•Run a nightly batch recomputation that becomes the billing source of truth.

Non-Functional Requirements

•Dashboard freshness within 1 minute of the click.
•Billing numbers must be reproducible and defensible to an advertiser.
•Ingest must absorb a 10x spike without dropping events.

Back-of-the-Envelope Math

1M clicks/s * 500 bytes = 500 MB/s ingest; a 7-day raw retention is ~300 TB.
Watermark of 5 minutes means a window is only finalized after late events have had time to land.

Key Architectural Trade-offs

The speed layer is fast and approximate, the batch layer slow and authoritative — running both is duplicated logic and the honest answer when money is involved.
Event-time windows with watermarks are correct for late data and delay finalization; processing-time windows are immediate and quietly wrong.
Deduplicating a billion events per hour demands bounded state — a time-scoped dedup window accepts duplicates older than the horizon.

Click or drag a component onto the canvas, then connect the handles to draw the data flow.

3 nodes · 2 edges

Components · 35

Client & Edge4

Compute & Gateway7

Storage & Caching11

Messaging & Streaming6

Coordination & Ops5

Intelligence2

React Flow