How would you design a Web Crawler at scale?

Question

Accepted Answer

## Step 1: Use Cases, Constraints, Assumptions, Back-of-Envelope

### Use cases
We'll scope to handle only the core flows that expose the bottlenecks; everything else is deferred.

- **Service** seeds frontier — enqueue seed URLs, normalize, dedupe, prioritize by PageRank/freshness.
- **Service** fetches — workers respect robots.txt, throttle per host (politeness 1 req/s), handle retries, detect duplicates via content hash.
- **User** queries crawl status — GET frontier depth, politeness stats, crawl DB snapshot.
- **Service** refreshes — schedule recrawl by change frequency (news hourly, docs daily).

### Out of scope
- Browser rendering/JS execution (separate headless pool), full search indexing inside crawler (indexer is downstream consumer), adversarial anti-bot bypass.

### State assumptions
- 1B pages to crawl, avg page 200 KB, revisit 30% daily (300M fetches/day).
- Fetch throughput needed: 300M/86k≈3,500 pages/s avg, 10k peak.
- Dedupe set 1B URLs (hash 8B → 8 GB Bloom filter,

How would you design a Web Crawler at scale?

The problem

Try it yourself

Solution

Step 1: Use Cases, Constraints, Assumptions, Back-of-Envelope

Use cases

Out of scope

State assumptions

The solution is hidden so you can try first

Discussion

Keep your momentum going.