Design Resumable Multi-Gigabyte Upload

Media & Large-Scale FilesMediumdesign-large-file-upload

Design Resumable Multi-Gigabyte Upload

Design uploads that survive a two-hour transfer on a flaky connection: parts uploaded in parallel, verified independently, and resumable from where the client stopped.

Multipart UploadChecksumsResumable SessionsBackpressure

Traffic & Capacity Estimates:

Files up to 5 TB · 50k concurrent uploads · resume after hours offline

Functional Requirements

•Initiate an upload session, upload parts independently and in parallel, then complete atomically.
•Resume after a disconnect without re-sending already-acknowledged parts.
•Verify each part with a checksum and the whole object on completion.
•Abort and garbage-collect abandoned sessions and their orphaned parts.

Non-Functional Requirements

•A part failure retries in isolation — never restarts the whole transfer.
•The object must not be visible or partially readable until completion.
•Storage for incomplete uploads must be reclaimed automatically.

Back-of-the-Envelope Math

A 5 TB file at 100 MB parts is 50,000 parts — part metadata is itself a data structure to manage.
50k concurrent uploads averaging 2 GB in flight = 100 TB of incomplete-upload storage at any moment.

Key Architectural Trade-offs

Larger parts mean less metadata and more re-sent bytes on a failure; smaller parts are resilient and multiply request count and bookkeeping.
Client-side parallelism saturates the link and can starve other traffic — backpressure and concurrency caps belong in the client contract.
Keeping abandoned parts costs storage; sweeping them too aggressively breaks genuinely slow resumes — the TTL is a real product decision.

Click or drag a component onto the canvas, then connect the handles to draw the data flow.

3 nodes · 2 edges

Components · 35

Client & Edge4

Compute & Gateway7

Storage & Caching11

Messaging & Streaming6

Coordination & Ops5

Intelligence2

React Flow