Design a Change Data Capture Pipeline

Data, Storage & ConsistencyMediumdesign-cdc-pipeline

Design the pipeline that keeps a search index, a cache, and a warehouse in step with the primary database — without any service dual-writing to both.

Change Data CaptureLog TailingDual-Write AvoidanceBackfill + Stream

Traffic & Capacity Estimates:

80k row changes/second · 4 downstream sinks · end-to-end lag under 5 seconds

•Capture inserts, updates, and deletes from the database write-ahead log, in commit order.
•Publish changes to a durable topic partitioned so per-row ordering is preserved.
•Backfill historical data and hand over to the live stream with no gap and no duplicate window.
•Let a sink be rebuilt independently by replaying from a chosen offset.

•End-to-end lag under 5 seconds at p99; lag must be measurable and alertable.
•At-least-once delivery with idempotent sink writes, so a replay is always safe.
•The capture process must not add write latency to the primary database.

80k changes/s * 1 KB = 80 MB/s into the log; 7-day retention = ~48 TB of broker storage.
A 30-minute connector outage accumulates ~144M events to catch up on — throughput headroom has to cover it.

Log-based CDC (complete, ordered, low overhead, tightly coupled to the database engine) vs query-based polling on updated_at (portable, misses deletes and fast successive updates).
CDC removes dual writes and replaces them with an asynchronous pipeline — the inconsistency doesn't vanish, it becomes a measurable lag you can alert on.
Retaining the log long enough for a full sink rebuild costs storage; retaining less means a rebuild needs a fresh snapshot and a careful handover.

Click or drag a component onto the canvas, then connect the handles to draw the data flow.

3 nodes · 2 edges

Components · 35

Client & Edge4

Compute & Gateway7

Storage & Caching11

Messaging & Streaming6

Coordination & Ops5

Intelligence2