MentorNode
Start free
Messaging & Event-DrivenMediumdesign-webhook-delivery

Design a Reliable Webhook Delivery Service

Design outbound event delivery to customer-controlled HTTP endpoints — the ones that are slow, occasionally down, sometimes malicious, and always someone else's problem to fix.

Retry with BackoffCircuit Breaker per EndpointSigned PayloadsDLQ
Traffic & Capacity Estimates:

300k events/second · 50k customer endpoints · retries over 24 hours

Functional Requirements

  • •Deliver each subscribed event to the customer's endpoint with a signed, verifiable payload.
  • •Retry failures on an exponential schedule for up to 24 hours, then dead-letter.
  • •Preserve per-subscription ordering where the customer requests it.
  • •Expose a delivery log and manual replay so customers can debug their own outages.

Non-Functional Requirements

  • •A dead or hanging endpoint must not consume delivery capacity for healthy ones.
  • •Outbound requests must be protected against SSRF into internal networks.
  • •At-least-once delivery, with enough metadata for the receiver to deduplicate.

Back-of-the-Envelope Math

  • 300k events/s with a 5% failure rate and 8 retries adds ~120k extra requests/second at steady state.
  • One endpoint timing out at 30s with 1,000 queued events pins 1,000 worker-seconds unless concurrency is capped per endpoint.

Key Architectural Trade-offs

  • Per-endpoint concurrency caps and circuit breakers isolate bad receivers; a shared worker pool is simpler and gets held hostage by the slowest one.
  • Strict ordering per subscription forces serial delivery and makes head-of-line blocking a customer-visible feature.
  • HMAC-signed payloads plus a timestamp prevent forgery and replay, at the cost of a real secret-rotation story.

Click or drag a component onto the canvas, then connect the handles to draw the data flow.

3 nodes · 2 edges

Components · 35

Client & Edge4
Compute & Gateway7
Storage & Caching11
Messaging & Streaming6
Coordination & Ops5
Intelligence2
Canvas overview