Design a RAG Pipeline over a Private Corpus

AI / ML InfrastructureHarddesign-rag-pipeline

Design a RAG Pipeline over a Private Corpus

Design retrieval-augmented generation over documents users own: ingestion and embedding, hybrid retrieval with reranking, and permission filtering that never leaks across tenants.

Chunking & EmbeddingHybrid RetrievalRerankingPer-Tenant Isolation

Traffic & Capacity Estimates:

100M documents · 2B chunks · 5k queries/second · answer within 3 seconds

Functional Requirements

•Ingest documents, chunk them with overlap, embed the chunks, and index them per tenant.
•Retrieve candidates with combined vector and keyword search, then rerank the top slice.
•Filter every retrieval by the requesting user's document permissions before generation.
•Re-embed and reindex incrementally when documents change or the embedding model is upgraded.

Non-Functional Requirements

•End-to-end answer latency under 3 seconds including generation.
•A tenant must never retrieve a chunk from another tenant — this is a hard isolation boundary.
•Retrieval quality must be measurable, with a regression suite gating model or chunking changes.

Back-of-the-Envelope Math

2B chunks * 768 dimensions * 4 bytes = ~6 TB of raw vectors; quantization to int8 brings it under 1.5 TB.
Re-embedding the whole corpus on a model upgrade is 2B inference calls — a migration project, not a config change.

Key Architectural Trade-offs

Hybrid retrieval (dense plus BM25) reliably beats either alone and doubles the retrieval infrastructure you have to operate.
Permission filtering before search needs per-tenant indexes or filtered ANN, both of which cost recall or cost isolation — post-filtering is simpler and leaks capacity to unauthorized results.
Larger chunks preserve context and dilute the embedding; smaller chunks retrieve precisely and lose the surrounding meaning — this is the main quality dial.

Click or drag a component onto the canvas, then connect the handles to draw the data flow.

3 nodes · 2 edges

Components · 35

Client & Edge4

Compute & Gateway7

Storage & Caching11

Messaging & Streaming6

Coordination & Ops5

Intelligence2

React Flow