Design a Web Crawler

Search, Ranking & DiscoveryHarddesign-web-crawler

Design a polite, distributed crawler that fetches billions of pages, never re-crawls the same URL twice, and revisits pages at a rate matched to how often they change.

Frontier QueuePoliteness SchedulingBloom Filter DedupDistributed BFS

Traffic & Capacity Estimates:

10B pages · 100k pages/second · 30-day recrawl cycle · 500 crawler nodes

Functional Requirements

•Maintain a prioritized URL frontier and extract new links from fetched pages.
•Respect robots.txt, crawl-delay, and per-domain politeness limits.
•Detect and skip already-seen URLs and near-duplicate content.
•Schedule recrawls based on observed change frequency per page.

Non-Functional Requirements

•Never overwhelm a single host regardless of how many URLs from it are queued.
•Crawler node failure must not lose the frontier or re-fetch large swathes of the web.
•Handle crawler traps, infinite calendars, and malicious redirect loops without stalling.

Back-of-the-Envelope Math

10B URLs in a dedup set: a Bloom filter at 1% false positives needs ~12 GB versus ~500 GB for the raw hashes.
100k pages/second * 100 KB average = 10 GB/second of ingest into storage and parsing.

Key Architectural Trade-offs

Partition the frontier by domain hash so politeness is enforced locally; partitioning by URL hash balances load and scatters per-domain rate limits everywhere.
Bloom filter dedup is memory-efficient and occasionally drops a legitimate page; an exact set is correct and needs a distributed store on the hot path.
Breadth-first crawling gets broad coverage; priority-driven crawling by PageRank or change rate gets a better index and risks starving the long tail.

Click or drag a component onto the canvas, then connect the handles to draw the data flow.

3 nodes · 2 edges

Components · 35

Client & Edge4

Compute & Gateway7

Storage & Caching11

Messaging & Streaming6

Coordination & Ops5

Intelligence2

React Flow