System Design (High-Level Design) Module 23 – Case studies: social, feeds and search

Design a web crawler

Design a web crawler from requirements and estimates to a polite URL frontier, URL and content deduplication, robots.txt rules and safe fetching.

Intermediate
25 minutes
Examples run with Python 3.14.8, Pyodide 314.0.7, Node.js 24.21.0 and quickjs 0.32.0
By MySmartCoPilot

Free: A search index needs breadth, freshness and good manners · The estimate: 386 pages a second need 1,160 busy hosts · Four records, keyed for the frontier: URL, host, page and link · 3 more sections. With a pass: 3 in-depth parts and 3 interview answers, about 16 minutes.

What you will learn

Estimate the pages a second, bandwidth, storage and connections of a crawl from its target
Design a URL frontier that obeys robots.txt and politeness while keeping fetchers busy
Detect duplicate URLs and near-duplicate content at the scale of billions of pages
Protect the crawler and the sites it visits, with limits on traps, addresses and response sizes

Before you start

What you’ll learn · Before you start

What you will learn

Estimate the pages a second, bandwidth, storage and connections of a crawl from its target
Design a URL frontier that obeys robots.txt and politeness while keeping fetchers busy
Detect duplicate URLs and near-duplicate content at the scale of billions of pages
Protect the crawler and the sites it visits, with limits on traps, addresses and response sizes

Before you start

On this page

A web crawler starts from a set of seed URLs, downloads each page, extracts its links and feeds the new ones back, until it has covered what it was asked to cover. At a billion pages a month, four parts share the work. A URL frontier decides which URL to fetch next and when each host may be contacted again. Fetchers download pages, caching DNS answers and each site’s robots.txt. A parser pulls out links and text. Two deduplication checks drop URLs seen before and pages already stored. What shapes the design is a speed limit: a polite crawler sends each host one request at a time, with a pause between, so its speed comes from how many hosts it keeps busy, not from its bandwidth. This case study sizes the crawler and follows its four paths.

A search index needs breadth, freshness and good manners

Ask what the crawl is for before drawing anything, because the answer sets every priority:

Purpose: a general search index, which needs breadth and freshness. An archive needs completeness instead, and a price monitor needs a few sites visited often.
Size and freshness: 1 billion pages a month, with popular pages fetched again every day.
Content: HTML pages. Other types are recorded by URL but not downloaded.
Manners: the public web only, obeying each site’s robots.txt and terms, with no logins and no form submissions.
Output: new and changed pages, published as events for an indexer.

The functional requirements: take seed URLs, fetch pages, extract and schedule their links, store pages with their metadata, fetch changed pages again, and report progress. The non-functional ones are what make it hard. The crawler fetches 386 pages a second on average and must be able to double that. It sends each host at most one request at a time, with a gap of at least a second, or longer when the site asks. It survives slow, broken and hostile servers, never fetches addresses on private networks, never loses a URL when a fetcher dies, and grows by adding machines.

The estimate: 386 pages a second need 1,160 busy hosts

Quantity	Assumption	Result
Pages a second	1 billion a month of 30 days	386 on average; plan for about 800
Download bandwidth	100 KB a page as fetched	38.6 MB a second, about 309 Mbit/s
Page storage	1 billion pages × 100 KB	100 TB a month before compression
Fetches in flight	2 seconds a fetch, by Little’s law	about 772, so about 800 open connections
URL-seen set	10 billion URLs known	80 GB as 8-byte fingerprints; about 12 GB as a Bloom filter at 1 % false positives
Hosts busy at once	each host: a 2-second fetch, then a 1-second gap	386 × 3 ≈ 1,160 hosts at every moment

The last row is the one most designs miss: more than a thousand hosts must have URLs waiting at every moment, or the fetchers sit idle while politeness holds them back. The Bloom filter: A small bit array built from a set of keys that answers "definitely not here" or "maybe here", never a false no. LSM engines keep one per file to skip the files that lack a key. More in “Storage engines: B-trees vs LSM-trees” row uses the sizing from Bloom and cuckoo filters, about 9.6 bits per item at 1 %.

Checkpoint

Check the estimate

Question 1 of 2 At 386 pages a second and 100 KB per page as fetched, how much download bandwidth does the crawler use, in megabits a second?
Type a number.
Your answer Mbit/s

A hint

Bytes a second first, then 8 bits a byte.

Show the answer: Question 1 of 2

Answer: 309 Mbit/s (anything from 278 to 340 counts)

386 × 100,000 bytes = 38.6 MB a second, and × 8 = about 309 megabits a second on average. Bandwidth is rarely the limit of a crawler; politeness towards each host is.
Question 2 of 2 A fetch takes 2 seconds on average, counting DNS, connecting and downloading. How many fetches are in flight at once at 386 pages a second?
Type a number.
Your answer fetches

A hint

Little's law multiplies the arrival rate by the time each one spends inside.

Show the answer: Question 2 of 2

Answer: 772 fetches (anything from 695 to 849 counts)

Little's law: requests in flight = arrival rate × time in the system = 386 × 2 ≈ 772. Each in-flight fetch holds a connection and a little memory, so the fleet needs about 800 concurrent connections, spread over many hosts.

Four records, keyed for the frontier: URL, host, page and link

The crawler is an internal service with a small control API:

POST /v1/crawls takes seeds, a scope, a priority and a recrawl policy, and returns a crawl id.
GET /v1/crawls/{id} reports progress: URLs known, fetched, failed and waiting.
GET /v1/pages?url=… returns the latest stored copy of a page and its metadata.
New and changed pages go out as events on a log, each with its URL, status, content fingerprint and the location of the body in the page store.

Record	Key	Holds
URL	fingerprint of the URL in normal form	the URL, its host, priority, status, when it is next due, and the ETag and Last-Modified of the last fetch
Host	host name	the robots.txt rules and when they expire, the gap between requests, when the host may next be contacted, its DNS answer and errors
Page	hash of the content	the compressed body in object storage, its headers, and the URLs that served it
Link	source and target URL	the link graph, kept for ranking later

A page travels four paths: schedule, fetch, parse and deduplicate

The crawler along its four paths: schedule, fetch, parse, deduplicate

Text description of the diagram

Seeds, sitemaps and the recrawl schedule (1) put URLs into the URL frontier, which keeps priorities and one queue per host.

The frontier (2) hands each fetcher a URL whose host may be contacted now. The fetcher, which caches DNS answers and robots.txt files, sends a GET to the web server and (3) writes the body and headers to the page store, compressed.

The parser (4) reads each stored page. Its text goes to the content check, which drops exact and near-duplicate pages and passes new pages on to the index and other consumers. Its links go to a step that normalises them and filters them by scope and limits, and then (5) to the URL-seen check, a Bloom filter in front of a store. Only URLs never seen before go back into the frontier.

Schedule. The frontier holds URLs by priority and hands a fetcher a URL only when that URL’s host has no request in flight and its gap has passed. Fetch. The fetcher resolves the host through its DNS cache, checks the cached robots.txt rules, sends the request with the validators stored from the last visit, and writes the response to the page store. Parse. The parser resolves each link against the page’s own URL and extracts the text. Deduplicate. Links are put in a One agreed spelling for all the ways of writing the same URL, so that two spellings compare equal: for example a lower-case scheme and host, no default port and no fragment., filtered by scope and limits, and checked against the URLs seen so far; only new ones return to the frontier. The page’s content is fingerprinted, so a page already stored under another address is recorded once and not parsed again.

The example below runs those four paths on a made-up web of two sites. It schedules first in, first out; priorities and politeness are the next step. Step through its first page, then run the whole crawl:

Interactive · stage by stage The first page of the crawl, path by path Step through one page, from the frontier to its three new links, with the checks that drop the fourth.

The drawing shows the whole scene; its 7 steps are listed below.

Every step as text

One seed. The crawl starts with one seed, https://books.example/, which is marked as seen at once, so no link can schedule it a second time.
Fetched. A fetcher takes the URL and downloads the page. In the example a table stands in for the network, so every run fetches the same pages.
A body never seen. The SHA-256 of the body matches no stored page, so this URL becomes the first one with that content, and the page goes on to the parser.
Four links. The parser finds /fiction/, /about#team, HTTPS://Books.Example:443/fiction/ and https://news.example/, and resolves each against the page's own URL.
One spelling each. Lower-case scheme and host, no default port, no fragment. /about#team becomes /about, and the capitalised address with :443 becomes https://books.example/fiction/.
One seen before. The first link has just scheduled https://books.example/fiction/, so its second spelling is dropped here. The other three links are new.
Back to the frontier. Only the three new URLs return to the frontier, first in, first out, so /fiction/ is fetched next. Eight more fetches finish the crawl.

Fetch, parse, deduplicate and schedule on a small made-up web Python · loop/crawl_loop.py

# The four paths of a crawler on a small made-up web: fetch a URL, parse its links, drop what was seen before (by
# URL and by content), and schedule the rest. FETCH stands in for the network, so every run prints the same.
import hashlib
from collections import deque
from html.parser import HTMLParser
from urllib.parse import urljoin, urlsplit, urlunsplit

FETCH = {  # URL -> (status, HTML)
    "https://books.example/": (200, '<a href="/fiction/">Fiction</a> <a href="/about#team">About us</a> '
                                    '<a href="HTTPS://Books.Example:443/fiction/">Fiction</a> <a href="https://news.example/">News</a>'),
    "https://books.example/fiction/": (200, '<h1>Fiction</h1> <a href="page-2">Next</a> <a href="../about">About</a> '
                                            '<a href="/fiction/?ref=home">Fiction</a> <a href="/missing">Old link</a>'),
    "https://books.example/fiction/?ref=home": (200, '<h1>Fiction</h1> <a href="page-2">Next</a> <a href="../about">About</a> '
                                                     '<a href="/fiction/?ref=home">Fiction</a> <a href="/missing">Old link</a>'),
    "https://books.example/fiction/page-2": (200, '<h1>Fiction, page 2</h1> <a href="/fiction/">Back</a>'),
    "https://books.example/about": (200, '<p>Write to us.</p> <a href="mailto:hello@books.example">Email</a> <a href="/">Home</a>'),
    "https://books.example/missing": (404, ""),
    "https://news.example/": (200, '<a href="/today">Today</a> <a href="http://news.example/today">Today (old address)</a> '
                                   '<a href="https://ads.example/track?id=7">Sponsored</a>'),
    "https://news.example/today": (200, "<h1>Today's stories</h1>"),
    "http://news.example/today": (200, "<h1>Today's stories</h1>"),
}
SCOPE = ("books.example", "news.example")  # hosts this crawl may visit


class Links(HTMLParser):
    def __init__(self):
        super().__init__()
        self.hrefs = []

    def handle_starttag(self, tag, attrs):
        if tag == "a":
            self.hrefs += [v for k, v in attrs if k == "href" and v]


def normalise(url):
    """A basic normal form: lower-case scheme and host, no default port, no fragment."""
    parts = urlsplit(url)
    host = parts.hostname or ""
    default = {"http": 80, "https": 443}.get(parts.scheme.lower())
    netloc = host if parts.port in (None, default) else f"{host}:{parts.port}"
    return urlunsplit((parts.scheme.lower(), netloc, parts.path or "/", parts.query, ""))


frontier = deque(["https://books.example/"])  # schedule: first in, first out here; priorities and politeness come later
seen_urls = set(frontier)
seen_bodies = {}  # SHA-256 of the body -> the first URL that had it
counts = {"fetched": 0, "links": 0, "seen before": 0, "out of scope": 0, "not http": 0, "same content": 0, "errors": 0}
while frontier:
    url = frontier.popleft()
    status, body = FETCH.get(url, (404, ""))  # fetch
    counts["fetched"] += 1
    if status != 200:
        counts["errors"] += 1
        print(f"{status}  {url}")
        continue
    digest = hashlib.sha256(body.encode()).hexdigest()  # deduplicate by content
    if digest in seen_bodies:
        counts["same content"] += 1
        print(f"200  {url}  same content as {seen_bodies[digest]}: stored as a duplicate, links not followed")
        continue
    seen_bodies[digest] = url
    parser = Links()  # parse
    parser.feed(body)
    new = []
    for href in parser.hrefs:
        counts["links"] += 1
        absolute = urljoin(url, href)
        if urlsplit(absolute).scheme not in ("http", "https"):
            counts["not http"] += 1
            continue
        link = normalise(absolute)
        if urlsplit(link).hostname not in SCOPE:
            counts["out of scope"] += 1
        elif link in seen_urls:  # deduplicate by URL
            counts["seen before"] += 1
        else:
            seen_urls.add(link)
            frontier.append(link)  # schedule
            new.append(link)
    print(f"200  {url}  {len(parser.hrefs)} link{'' if len(parser.hrefs) == 1 else 's'}, {len(new)} new")

print()
print(", ".join(f"{k}: {v}" for k, v in counts.items()))
print(f"{len(seen_urls)} distinct URLs scheduled for {len(seen_bodies)} distinct pages of content")

Output

200  https://books.example/  4 links, 3 new
200  https://books.example/fiction/  4 links, 3 new
200  https://books.example/about  2 links, 0 new
200  https://news.example/  3 links, 2 new
200  https://books.example/fiction/page-2  1 link, 0 new
200  https://books.example/fiction/?ref=home  same content as https://books.example/fiction/: stored as a duplicate, links not followed
404  https://books.example/missing
200  https://news.example/today  0 links, 0 new
200  http://news.example/today  same content as https://news.example/today: stored as a duplicate, links not followed

fetched: 9, links: 14, seen before: 4, out of scope: 1, not http: 1, same content: 2, errors: 1
9 distinct URLs scheduled for 6 distinct pages of content

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 crawl_loop.py

Nine fetches found 14 links: four already scheduled, one outside the scope and one email address. Two pages repeated content stored under another URL, one behind a tracking parameter and one at an old http:// address. The basic normal form merged HTTPS://Books.Example:443/fiction/ with /fiction/; the exercise completes it with the rules of RFC 3986.

Checkpoint

Check: Schedule, fetch, parse and deduplicate

Question 1 of 2 Which checks does a newly found link pass before it enters the URL frontier?
Choose every answer that is right.

Its host's robots.txt is downloaded again Its host is inside the crawl's scope The URL-seen check says it has not been scheduled before It is put in a normal form, so two spellings of one address count once The page it points to is compared with stored pages for near-duplicates
Show the answer: Question 1 of 2
Answer:
- Its host is inside the crawl's scope
- The URL-seen check says it has not been scheduled before
- It is put in a normal form, so two spellings of one address count once
Links are normalised, filtered by scope and limits, and checked against the URLs seen so far. Comparing content needs the page itself, so it happens after the fetch, and robots.txt is fetched once per host and cached, not per link.
Question 2 of 2 In the run, https://books.example/fiction/?ref=home returned the same body as /fiction/. What did the crawler do with it?
Choose one answer.

Refused to fetch it, because its URL has a query string Parsed it and scheduled its links a second time Stored it as a duplicate of /fiction/ and did not follow its links Deleted the stored copy of /fiction/

Show the answer: Question 2 of 2

Answer: Stored it as a duplicate of /fiction/ and did not follow its links

The content check runs after the fetch and before the parse: a body whose fingerprint is already stored is recorded as a duplicate of the first URL that had it, and its links are not followed.

Practise: put every link in its normal form

The exercise asks for the full normal form of RFC 3986, and its sample tests run in the page:

Exercise · Medium · Python

Put URLs in a normal form before the seen check

Write normal_form(url) in normal_form.py. A crawler calls it on every link before the URL-seen check, so that two spellings of the same address count as one. Follow the normalisations of RFC 3986, section 6:

the scheme and the host in lower case; the path and the query keep their case;
no port when it is the scheme's default (80 for http, 443 for https); any other port stays;
an empty path becomes /, and the dot segments . and .. are removed from the path (/a/./b/../c is /a/c);
percent-encodings use upper-case hex digits (%3a becomes %3A), and those that encode an unreserved character (a letter, a digit, -, ., _ or ~) are decoded (%7E becomes ~);
no fragment: everything from # on is dropped.

The query string stays as it is. Changing it (sorting or dropping parameters) can change which page a server returns, so it is a decision for each site, not for every URL.

Starter code · normal_form.py

import re
from urllib.parse import urlsplit, urlunsplit

UNRESERVED = set("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~")


def normal_form(url):
    """The URL in the normal form of RFC 3986, section 6, without its fragment."""
    # Replace this line with your code.
    return url

The sample tests · test_normal_form.py

from normal_form import normal_form


def test_scheme_and_host_lower_case_path_kept():
    """only the scheme and the host are case-insensitive"""
    assert normal_form("HTTP://Books.Example/Fiction/Page") == "http://books.example/Fiction/Page"


def test_default_ports_dropped_others_kept():
    """:443 is https's default, :8080 is not"""
    assert normal_form("https://books.example:443/a") == "https://books.example/a"
    assert normal_form("http://books.example:80/a") == "http://books.example/a"
    assert normal_form("http://books.example:8080/a") == "http://books.example:8080/a"


def test_dot_segments_removed():
    """. and .. are resolved inside the path"""
    assert normal_form("https://books.example/a/./b/../c") == "https://books.example/a/c"
    assert normal_form("https://books.example/a/b/../../c/") == "https://books.example/c/"


def test_empty_path_and_fragment():
    """an empty path is /, and the fragment never reaches the server"""
    assert normal_form("https://books.example") == "https://books.example/"
    assert normal_form("https://books.example/a#reviews") == "https://books.example/a"


def test_percent_encodings():
    """unreserved characters are decoded; other triplets get upper-case hex digits"""
    assert normal_form("https://books.example/%7euser/a%3ab") == "https://books.example/~user/a%3Ab"
    assert normal_form("https://books.example/%41%2F") == "https://books.example/A%2F"


def test_query_kept_as_it_is():
    """parameters are neither sorted nor dropped"""
    assert normal_form("https://books.example/s?q=Dosa&page=2") == "https://books.example/s?q=Dosa&page=2"

A hint

Split the URL with urlsplit, and rebuild it with urlunsplit((scheme, netloc, path, query, "")). For the dot segments, walk path.split("/"): skip ".", and for ".." remove the last segment kept so far. For the percent-encodings, re.sub(r"%[0-9A-Fa-f]{2}", fix, text) calls fix with each match: return the decoded character when it is unreserved, otherwise the match in upper case.

The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.

Then practise the four ideas above, starting with the one you have answered least well:

The deep dives this design must answer

The first complete design above leaves these questions for the second half of a round:

Politeness at full speed: priority queues in front and one queue per host behind, so a busy host never blocks the others.
Robots rules: RFC 9309 defines allow and disallow rules for each crawler, and what a failing robots.txt means.
DNS: a cache keyed by host saves most lookups, because one host’s URLs are fetched together.
Recrawling: conditional requests ask a server to send a page only if it changed.
Near-duplicates: SimHash gives similar pages fingerprints that differ in only a few bits.
Traps: limits per host, path pattern and parameter stop sites that make up addresses without end.
Safety: address checks after DNS keep the crawler away from private networks.
Scale: the bottlenecks move at 10 × and 100 × the pages.

Interviewers usually pick two or three of them, so a strong answer can go deep on any of them.

Politeness, robots rules and the frontier, measured Pro: opens with a Pro, Premium or Ultimate pass

A crawler’s speed is set by how politely it treats each host, so the frontier is the heart of the design. This part applies robots.txt the way RFC 9309 defines it, runs a polite frontier on a simulated clock, and turns the result into the number of hosts a real crawl must keep busy.

In this part:

What robots.txt decides, rule by rule
The frontier: priority queues in front, one queue per host behind
Running it: four fetchers, three hosts, measured
How many hosts must be busy, with numbers
DNS, the hidden bottleneck
Recrawling changed pages for less

1 diagram · 2 runnable examples (Python) · about 16 minutes

Opening this part with your pass…

This part opens with any MySmartCoPilot pass.

Pro pass · 1 week₹79$2.99
30 days₹179$5.99
1 year₹1,699$45

One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.

Choose a pass I have a licence key

It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).

Read a complete sample first: Design a news feed

Near-duplicates, traps and safe fetching Pro: opens with a Pro, Premium or Ultimate pass

Two problems waste most of a careless crawler’s budget: the same content under many addresses, and sites that make up new addresses without end. A third, fetching addresses it should never touch, is a security hole. This part measures the first and sets limits for the other two.

In this part:

Near-duplicate pages with SimHash, measured
Crawler traps and the limits that stop them
Fetching safely: addresses, redirects and sizes
Failure drills, from a dead fetcher to a lost frontier shard

1 diagram · 1 runnable example (JavaScript) · about 8 minutes

Opening this part with your pass…Pro part. It opens with the same pass as the part above.

Bottlenecks at 10× and 100×, the level rubric and a self-mock Pro: opens with a Pro, Premium or Ultimate pass

The last part scales the design up, lists the alternatives it rejects, and turns it into an answer you can give in a round. Use the rubric to score your own answer.

In this part:

Bottlenecks at 10× and 100×, with numbers
Five simpler designs, and why each fails
What each level is expected to cover
A sample answer in ten minutes
A 45-minute self-mock

about 4 minutes

Opening this part with your pass…Pro part. It opens with the same pass as the part above.

Interview questions

Fresher to mid · Warm-up

What does robots.txt control, and what does it not?

It tells crawlers which URLs of a site they may fetch: the site publishes groups of allow and disallow rules for named crawlers, and RFC 9309 standardises how crawlers read them. It is a request, not a lock. The RFC says the rules are not a form of access authorization, so a crawler that ignores them can still fetch the pages, and listing a private path in robots.txt makes it public. It also controls fetching, not what a search engine shows: Google's documentation says robots.txt is not a way to keep a page out of its results, which takes a noindex rule or a password. Use robots.txt to manage crawler traffic, and real authentication to protect content.
Mid to senior

A large news site's robots.txt has answered 503 for a day, while its pages load normally. What does your crawler do today, after a week and after a month, and why?

Model answer, Pro. Model answers open with any pass; prices are in the notice on this page. First step and follow-ups
Mid to senior

Half the URLs in your frontier belong to one site of 50 million pages, which allows one request every 2 seconds. Design the frontier so that the crawler stays busy and the big site is still covered sensibly.

Model answer, Pro. First step and follow-ups
Senior to staff

A crawl of a shopping site produces millions of URLs that differ only in sort, filter and session parameters, and most of them show the same products. How do you detect this and stop it without missing real pages?

Model answer, Pro. First step and follow-ups

Key takeaways

A crawler is four paths: schedule in a frontier, fetch politely, parse links and text, and deduplicate URLs and content before scheduling again.
Politeness sets the speed: at 386 pages a second about 1,160 hosts must be busy at once.
Put every link in a normal form before the URL-seen check, and fingerprint every page before parsing it again.
Size the URL-seen set early: 10 billion URLs are 80 GB of fingerprints or about 12 GB of Bloom filter.

References

RFC 9309: Robots Exclusion Protocol (IETF)
Sitemaps XML format (sitemaps.org)
Overview of Google crawlers and fetchers (Google Search Central)
Introduction to robots.txt (Google Search Central)
Detecting near-duplicates for web crawling (WWW conference (Manku, Jain and Das Sarma))
Server-Side Request Forgery Prevention Cheat Sheet (OWASP)
RFC 3986: Uniform Resource Identifier (URI): Generic Syntax, section 6 (normalization and comparison) (IETF)
RFC 9110: HTTP Semantics, section 13.1 (preconditions) and section 15.4.5 (304 Not Modified) (IETF)
urllib.parse (parse URLs into components) (Python Software Foundation)

Report a problem with this lesson

In this module: Case studies: social, feeds and search

Design a news feed Free: 35 minutes
Design X (Twitter): timelines and search Free core, Pro depth: 9 of 30 minutes free
Design search autocomplete Free intro, Pro: 3 of 15 minutes free
Design a web crawler Free core, Pro depth: 9 of 25 minutes free
Design a simplified web search engine Free intro, Pro: 4 of 25 minutes free
Design a Q&A forum like Quora or Reddit Free intro, Pro: 5 of 25 minutes free
Design a news aggregator Free intro, Pro: 5 of 25 minutes free
Design dating matches like Tinder Free core, Pro depth: 8 of 25 minutes free
Design top-K trending topics Free core, Pro depth: 9 of 25 minutes free

Your country

Design a web crawler

What you will learn

Before you start

What you will learn

Before you start

A search index needs breadth, freshness and good manners

The estimate: 386 pages a second need 1,160 busy hosts

Four records, keyed for the frontier: URL, host, page and link

A page travels four paths: schedule, fetch, parse and deduplicate

Practise: put every link in its normal form

Put URLs in a normal form before the seen check

Results of the sample tests

The deep dives this design must answer

Politeness, robots rules and the frontier, measured Pro: opens with a Pro, Premium or Ultimate pass

Near-duplicates, traps and safe fetching Pro: opens with a Pro, Premium or Ultimate pass

Bottlenecks at 10× and 100×, the level rubric and a self-mock Pro: opens with a Pro, Premium or Ultimate pass

Interview questions

What does robots.txt control, and what does it not?

A large news site's robots.txt has answered 503 for a day, while its pages load normally. What does your crawler do today, after a week and after a month, and why?

Half the URLs in your frontier belong to one site of 50 million pages, which allows one request every 2 seconds. Design the frontier so that the crawler stays busy and the big site is still covered sensibly.

A crawl of a shopping site produces millions of URLs that differ only in sort, filter and session parameters, and most of them show the same products. How do you detect this and stop it without missing real pages?

Key takeaways

References

Related tools