Design a Feature Store

AI / ML InfrastructureMediumdesign-feature-store

Design the store that serves the same feature values to training and inference — the layer whose absence produces models that score well offline and fail in production.

Online/Offline ParityPoint-in-Time CorrectnessStreaming FeaturesBackfill

Traffic & Capacity Estimates:

5k features · 500M entities · 200k online reads/second · training sets over years of history

Functional Requirements

•Define features once and materialize them to both an online store and an offline store.
•Serve low-latency online lookups by entity key for inference.
•Generate point-in-time-correct training sets that never include data from after the label event.
•Support batch, streaming, and on-demand computed features under the same definition.

Non-Functional Requirements

•Online read p99 under 10ms for a batch of 200 features.
•Online and offline values for the same entity and timestamp must agree exactly.
•Feature freshness must be monitored — silent staleness is a silent model regression.

Back-of-the-Envelope Math

500M entities * 5k features is far too wide to materialize fully — only serving-relevant feature groups go online.
200k reads/s * 200 features = 40M feature values/second from the online store.

Key Architectural Trade-offs

One definition compiled to two execution paths (batch and streaming) prevents training/serving skew and is substantially harder to build than two pipelines.
Point-in-time-correct joins are the difference between an honest offline metric and label leakage, and they make training-set generation expensive.
Fresher streaming features improve model quality and add a real-time pipeline to the critical path of every prediction.

Click or drag a component onto the canvas, then connect the handles to draw the data flow.

3 nodes · 2 edges

Components · 35

Client & Edge4

Compute & Gateway7

Storage & Caching11

Messaging & Streaming6

Coordination & Ops5

Intelligence2

React Flow