Design a Recommendation Serving Service

Search, Ranking & DiscoveryHarddesign-recommendation-service

Design a Recommendation Serving Service

Design the serving path for recommendations: narrow millions of items to a few hundred candidates, rank them with a model, and do it in under 100ms per request.

Candidate Generation + RankingEmbedding RetrievalFeature StoreOnline/Offline Split

Traffic & Capacity Estimates:

200M users · 50M items · 100k recommendation requests/second · p99 under 100ms

Functional Requirements

•Generate candidates from several sources (collaborative, content-based, trending, recent).
•Rank candidates with a model using fresh user and item features.
•Apply business rules — diversity, freshness, already-seen suppression, policy filters.
•Log impressions and outcomes for model retraining and online evaluation.

Non-Functional Requirements

•End-to-end p99 under 100ms including feature fetch and model inference.
•Model and feature versions must stay in lockstep — training/serving skew is a correctness bug.
•A model-serving outage must fall back to a non-personalized ranking, not an empty feed.

Back-of-the-Envelope Math

100k requests/s * 500 candidates = 50M item scorings per second — batched inference is the only way this is affordable.
Feature fetch of 200 features per request at 100k RPS = 20M feature reads/second from the online store.

Key Architectural Trade-offs

Multi-stage retrieval (cheap ANN recall, then expensive ranking on a few hundred) is what makes a heavy model affordable; single-stage scoring does not scale past small catalogs.
Precomputed recommendations per user are cheap to serve and stale by hours; on-request generation is fresh and costs inference on every view.
A shared feature store guarantees online/offline parity and becomes a hard dependency on the serving path — its latency is your latency.

Click or drag a component onto the canvas, then connect the handles to draw the data flow.

3 nodes · 2 edges

Components · 35

Client & Edge4

Compute & Gateway7

Storage & Caching11

Messaging & Streaming6

Coordination & Ops5

Intelligence2

React Flow