Design Service Discovery & Leader Election

Foundations & Core PatternsHarddesign-service-discovery

Design Service Discovery & Leader Election

Design the cluster membership layer: services find each other as instances come and go, and exactly one instance holds a leadership lease at any moment.

Consensus (Raft)Leader ElectionHealth CheckingLease & Heartbeat

Traffic & Capacity Estimates:

20,000 instances · 5-node consensus cluster · sub-10s failure detection

Functional Requirements

•Instances register on start and deregister on shutdown; callers resolve a service name to healthy endpoints.
•Detect dead instances via heartbeats/leases and evict them from the endpoint set automatically.
•Elect a single leader for singleton workloads (schedulers, compaction, migrations) with a fencing token.
•Push membership changes to watchers rather than requiring them to poll.

Non-Functional Requirements

•Resolution must survive the discovery cluster being unreachable — clients cache last-known-good endpoints.
•Leadership safety over liveness: two simultaneous leaders is a correctness bug, a brief gap with none is not.
•Failure detection within 10 seconds without false-positive evictions during GC pauses.

Back-of-the-Envelope Math

20,000 instances heartbeating every 3s = ~6,700 writes/second into the consensus store.
A 5-node Raft cluster tolerates 2 failures; write throughput is bounded by the slowest quorum member.

Key Architectural Trade-offs

Consensus-backed registry (ZooKeeper/etcd — strongly consistent, write-throughput bound) vs gossip-based membership (scales further, eventually consistent view).
Client-side discovery with caching (no extra hop, thick clients) vs server-side via a load balancer or mesh (thin clients, another tier to run).
Leases prevent split-brain only if the work itself is fenced — an old leader that resumes after a GC pause must be rejected by the downstream store, not just by the lease.

Click or drag a component onto the canvas, then connect the handles to draw the data flow.

3 nodes · 2 edges

Components · 35

Client & Edge4

Compute & Gateway7

Storage & Caching11

Messaging & Streaming6

Coordination & Ops5

Intelligence2

React Flow