Design a Reliable Webhook Delivery Service

Messaging & Event-DrivenMediumdesign-webhook-delivery

Design a Reliable Webhook Delivery Service

Design outbound event delivery to customer-controlled HTTP endpoints — the ones that are slow, occasionally down, sometimes malicious, and always someone else's problem to fix.

Retry with BackoffCircuit Breaker per EndpointSigned PayloadsDLQ

Traffic & Capacity Estimates:

300k events/second · 50k customer endpoints · retries over 24 hours

Functional Requirements

•Deliver each subscribed event to the customer's endpoint with a signed, verifiable payload.
•Retry failures on an exponential schedule for up to 24 hours, then dead-letter.
•Preserve per-subscription ordering where the customer requests it.
•Expose a delivery log and manual replay so customers can debug their own outages.

Non-Functional Requirements

•A dead or hanging endpoint must not consume delivery capacity for healthy ones.
•Outbound requests must be protected against SSRF into internal networks.
•At-least-once delivery, with enough metadata for the receiver to deduplicate.

Back-of-the-Envelope Math

300k events/s with a 5% failure rate and 8 retries adds ~120k extra requests/second at steady state.
One endpoint timing out at 30s with 1,000 queued events pins 1,000 worker-seconds unless concurrency is capped per endpoint.

Key Architectural Trade-offs

Per-endpoint concurrency caps and circuit breakers isolate bad receivers; a shared worker pool is simpler and gets held hostage by the slowest one.
Strict ordering per subscription forces serial delivery and makes head-of-line blocking a customer-visible feature.
HMAC-signed payloads plus a timestamp prevent forgery and replay, at the cost of a real secret-rotation story.

Click or drag a component onto the canvas, then connect the handles to draw the data flow.

3 nodes · 2 edges

Components · 35

Client & Edge4

Compute & Gateway7

Storage & Caching11

Messaging & Streaming6

Coordination & Ops5

Intelligence2

React Flow