Design a Vector Search / Embedding Index

AI / ML InfrastructureHarddesign-vector-search

Design a Vector Search / Embedding Index

Design approximate nearest-neighbour search at billion-vector scale: index structure, recall against latency, quantization, and the awkwardness of updating an ANN index in place.

ANN Indexing (HNSW/IVF)QuantizationIndex ShardingIncremental Updates

Traffic & Capacity Estimates:

2B vectors · 768 dimensions · 20k queries/second · recall@10 above 0.95 · p99 under 50ms

Functional Requirements

•Index high-dimensional vectors and return the k nearest neighbours for a query vector.
•Support metadata filters applied alongside the vector search.
•Insert, update, and delete vectors without a full index rebuild.
•Shard the index across nodes and merge results from each shard.

Non-Functional Requirements

•p99 query latency under 50ms with recall@10 of at least 0.95.
•Memory footprint must fit the fleet — raw float32 vectors do not.
•Index build and merge must run without taking the index offline.

Back-of-the-Envelope Math

2B * 768 * 4 bytes = 6 TB raw; product quantization at 8x compression brings it to ~750 GB.
HNSW graph overhead adds roughly 30-50% on top of the vector data itself.

Key Architectural Trade-offs

HNSW gives excellent recall and latency with high memory cost and awkward deletes; IVF-PQ is memory-frugal and needs careful probe tuning to reach the same recall.
Quantization is what makes billion-scale affordable and costs recall — the compression ratio is a direct quality dial.
Filtered vector search is genuinely hard: pre-filtering breaks the graph's connectivity assumptions, post-filtering can return nothing after the filter is applied.

Click or drag a component onto the canvas, then connect the handles to draw the data flow.

3 nodes · 2 edges

Components · 35

Client & Edge4

Compute & Gateway7

Storage & Caching11

Messaging & Streaming6

Coordination & Ops5

Intelligence2

React Flow