MentorNode
Start free
Media & Large-Scale FilesMediumdesign-image-pipeline

Design an Image Upload & Transcoding Pipeline

Design the path from a user's camera roll to a fast-loading image: direct upload, validation, derivative generation, format negotiation, and edge delivery.

Presigned UploadAsync Worker PoolOn-the-Fly DerivativesEdge Caching
Traffic & Capacity Estimates:

200M uploads/day · 6 derivatives each · 20B image reads/day · p95 render under 200ms

Functional Requirements

  • •Upload directly to blob storage via presigned URLs, bypassing application servers.
  • •Validate content type, dimensions, and safety before the image becomes publicly visible.
  • •Generate derivatives (thumbnails, feed sizes, WebP/AVIF variants) and strip EXIF metadata.
  • •Serve the best format and size for the requesting device from the edge.

Non-Functional Requirements

  • •The uploader sees their image immediately, even while derivatives are still processing.
  • •A transcoding backlog must not block uploads.
  • •Malicious files (decompression bombs, polyglots) must be rejected before processing.

Back-of-the-Envelope Math

  • 200M uploads/day * 6 derivatives = 1.2B derivative jobs/day (~14k/second average).
  • At 300ms CPU per derivative, 14k/s needs ~4,200 always-on worker cores before peak headroom.

Key Architectural Trade-offs

  • Eager derivative generation makes the first view fast and burns compute on images nobody opens; on-the-fly resizing behind the CDN pays only for what's requested and adds cold-miss latency.
  • Presigned direct upload removes the bytes from your API tier and forces validation to happen after the object already exists.
  • Serving AVIF/WebP cuts bandwidth substantially and multiplies the derivative matrix — format negotiation has to happen at the edge to stay cacheable.

Click or drag a component onto the canvas, then connect the handles to draw the data flow.

3 nodes · 2 edges

Components · 35

Client & Edge4
Compute & Gateway7
Storage & Caching11
Messaging & Streaming6
Coordination & Ops5
Intelligence2
Canvas overview
React Flow