Design Cache Invalidation for a Read-Heavy API

Caching & Content DeliveryMediumdesign-cache-invalidation

Design Cache Invalidation for a Read-Heavy API

Design the read path for an API with a 100:1 read-to-write ratio, where the hard part is not caching — it's deciding when a cached value stops being true.

Cache-AsideThundering Herd ControlWrite-ThroughTTL Jitter

Traffic & Capacity Estimates:

200k reads/second · 2k writes/second · staleness budget of 60 seconds

Functional Requirements

•Serve reads from cache and populate on miss, keyed by the query shape and the caller's visibility scope.
•Invalidate or refresh affected entries when the underlying record is written.
•Collapse concurrent misses for the same key into a single origin fetch.
•Support an explicit bypass for callers that need read-your-own-writes.

Non-Functional Requirements

•Cache hit ratio above 95% in steady state.
•No more than a 60-second window where a user sees stale data after someone else's write.
•A mass eviction (deploy, restart, flush) must not take the database down with it.

Back-of-the-Envelope Math

200k reads/s at 95% hit ratio leaves 10k reads/s hitting the database.
One expired hot key with 5,000 concurrent readers becomes 5,000 identical DB queries — unless requests are collapsed.

Key Architectural Trade-offs

Invalidate on write (tight, fails silently when a write path is missed) vs short TTL (self-healing, permanently stale within the window).
Thundering herd control: a per-key mutex/single-flight lock, probabilistic early refresh, or serving stale while revalidating in the background.
Uniform TTLs cause synchronized expiry cliffs; jittered TTLs smooth the origin load at the cost of less predictable freshness.

Click or drag a component onto the canvas, then connect the handles to draw the data flow.

3 nodes · 2 edges

Components · 35

Client & Edge4

Compute & Gateway7

Storage & Caching11

Messaging & Streaming6

Coordination & Ops5

Intelligence2

React Flow