MentorNode
Start free
Platform & InfrastructureHarddesign-log-aggregation

Design Distributed Log Aggregation & Tracing

Design centralized logs and traces for a microservice fleet: ship, buffer, index, and search terabytes a day, with trace context that survives every network hop.

Log ShippingDistributed TracingSamplingIndex vs Object Storage
Traffic & Capacity Estimates:

10 TB logs/day · 5B spans/day · search over 7 days in under 5 seconds

Functional Requirements

  • •Ship structured logs from every host with local buffering that survives a backend outage.
  • •Propagate trace and span context across service boundaries and assemble complete traces.
  • •Search and filter by service, time range, trace ID, and arbitrary fields.
  • •Tier old data to cheap object storage while keeping it queryable.

Non-Functional Requirements

  • •Log shipping must never block or crash the application producing the logs.
  • •Ingest must absorb an incident-driven 10x log spike — which is exactly when you need it.
  • •Trace sampling decisions must be consistent across all services in one request.

Back-of-the-Envelope Math

  • 10 TB/day fully indexed can cost more in index storage than in raw data — indexing choices dominate the bill.
  • 5B spans/day at 1% head sampling stores 50M spans/day; tail sampling keeps the interesting 0.1% instead.

Key Architectural Trade-offs

  • Head-based sampling is cheap and decided before you know whether the request was interesting; tail-based sampling keeps the errors and requires buffering every span until the trace completes.
  • Full-text indexing every field makes search instant and multiplies cost; indexing a few fields and scanning object storage for the rest is far cheaper and slower.
  • The logging pipeline is load-bearing during incidents, so it must have its own capacity headroom and its own failure domain.

Click or drag a component onto the canvas, then connect the handles to draw the data flow.

3 nodes · 2 edges

Components · 35

Client & Edge4
Compute & Gateway7
Storage & Caching11
Messaging & Streaming6
Coordination & Ops5
Intelligence2
Canvas overview
React Flow