MentorNode
Start free
Search, Ranking & DiscoveryMediumdesign-trending-topics

Design Trending Topics (Top-K Heavy Hitters)

Design real-time trend detection over a firehose: find the top 100 rising terms in the last 5 minutes without counting every term exactly.

Count-Min SketchSliding WindowsStream ProcessingApproximate Top-K
Traffic & Capacity Estimates:

1M events/second · millions of distinct terms · 5-minute window · updated every 10s

Functional Requirements

  • •Maintain approximate counts for terms over sliding time windows.
  • •Rank by velocity (rate of change) rather than raw volume, so perennial terms don't dominate.
  • •Segment trends by geography and language.
  • •Suppress spam, bot amplification, and blocklisted terms before publication.

Non-Functional Requirements

  • •Trends refresh at least every 10 seconds.
  • •Memory per window must be bounded regardless of term cardinality.
  • •A processing node failure must lose at most one window, not corrupt history.

Back-of-the-Envelope Math

  • Exact counting of 50M distinct terms per window needs tens of GB; a Count-Min sketch holds it in ~50 MB with a small error bound.
  • 1M events/second * 300s window = 300M term occurrences in flight at any moment.

Key Architectural Trade-offs

  • Approximate sketches (bounded memory, overcounts rare terms) vs exact counts (perfectly right, memory grows with cardinality) — for top-K, approximation is nearly free.
  • Tumbling windows are simple and produce jumpy trends; sliding windows are smooth and multiply the state kept per term.
  • Velocity ranking surfaces genuinely breaking topics and makes the system far easier to manipulate with coordinated bursts.

Click or drag a component onto the canvas, then connect the handles to draw the data flow.

3 nodes · 2 edges

Components · 35

Client & Edge4
Compute & Gateway7
Storage & Caching11
Messaging & Streaming6
Coordination & Ops5
Intelligence2
Canvas overview
React Flow