Web Scraping & Browser Automation (Python and JavaScript) Module 1 – Permission first: robots.txt, terms, law and limits
robots.txt for your bot: RFC 9309 rules
Read robots.txt the way RFC 9309 does: find your bot's group, let the longest rule decide, act on 4xx, 429 and 5xx answers, and pick a parser that agrees.
What you will learn
- Find the group that applies to your product token, and decide each path with the longest matching rule
- Handle every answer to the robots.txt request itself, from a success and redirects to 4xx, 429, 5xx and network errors
- Reuse a copy of robots.txt for at most 24 hours per site, and keep an older copy only while the file is unreachable
- Choose a robots.txt parser whose answers match RFC 9309 on the Python you run
Before you start
What you’ll learn · Before you start
What you will learn
- Find the group that applies to your product token, and decide each path with the longest matching rule
- Handle every answer to the robots.txt request itself, from a success and redirects to 4xx, 429, 5xx and network errors
- Reuse a copy of robots.txt for at most 24 hours per site, and keep an older copy only while the file is unreachable
- Choose a robots.txt parser whose answers match RFC 9309 on the Python you run
Before you start
On this page
Put twelve questions to the practice shop’s robots.txt, and RFC 9309, the standard that defines the file, and the
urllib.robotparser of Python 3.14.2, the version inside the browser runner, disagree on six of them. A collector that
trusts the wrong reading walks into pages the shop asked bots to avoid, or skips pages it may fetch.
Here you read robots.txt the way the standard does: find the one group that applies to your bot’s product token, let the longest matching rule decide each path, act on every kind of answer to the robots.txt request itself, keep a copy for at most a day, and choose a parser only after checking its answers against cases like the twelve below.
robots.txt is a request, read before anything else
robots.txt: A plain text file at /robots.txt in which a site asks automated clients to stay out of some paths, group by product token. RFC 9309 defines it; it is a request, not access control. is a plain text file in which a site lists the parts it asks crawlers to leave alone. RFC 9309 defines it, and says in its first section that these rules are not a form of access authorization: the file asks, it does not lock anything. The same standard warns that listing a path in robots.txt makes it public and easy to discover, so the file is never a list of secrets (RFC 9309, sections 1 and 3).
The file lives at one address per service, /robots.txt in lower case at the top of the path, in UTF-8 with the media
type text/plain (section 2.3). It speaks only for the scheme, host and port it is served from, as Google’s
documentation spells out: the rules at https://bazaar.nimbu.test/robots.txt say nothing about another subdomain
(Google’s robots.txt documentation). Here
is the practice shop’s file as this lesson serves it, with three groups for named bots beside its * group; the rest
of this lesson reads it:
# Nimbu Bazaar, the practice shop: every product here is invented.
User-agent: *
Disallow: /account/
Disallow: /people/
Disallow: /products/*?sort=
Allow: /products/
Crawl-delay: 2
User-agent: NimbuBot
Disallow: /prices/
Allow: /prices/today
Disallow: /*.pdf$
Disallow: /offers/
Allow: /offers/
Crawl-delay: 2.5
User-agent: nimbubot
Disallow: /drafts/
User-agent: ArchiveBot
Disallow: /
Sitemap: https://bazaar.nimbu.test/sitemap.xmlIt has four groups. Each starts with one or more User-agent lines and holds Allow and Disallow rules; the
Crawl-delay: A robots.txt line that asks a bot to wait a number of seconds between requests. It is not part of RFC 9309, and parsers differ in how they read it. and Sitemap lines are other records, which the standard lets parsers
read or ignore. Follow a polite bot through its first minute on the shop:
The drawing shows the whole scene; its 7 steps are listed below.
Every step as text
- The rules first. Before any page, the bot asks for /robots.txt at the top of the shop's address, with a User-Agent that names it and gives a contact.
- The file arrives. The shop answers 200 with a text/plain file in UTF-8: four groups of rules and a Sitemap line. The status matters as much as the text, as a later section shows.
- Its own group. The bot finds the groups that name its product token. NimbuBot and nimbubot are the same token in any case, so both groups merge into one with six rules.
- An allowed page. /prices/today matches two rules, and the longer one, Allow: /prices/today, decides. The bot sends the request.
- A request never sent. /prices/history matches only Disallow: /prices/, so the bot does not ask for it at all. Nothing reaches the shop, which is the point of reading the file first.
- No rule, so allowed. No rule of NimbuBot's group matches a product page, and a path that no rule matches is allowed. The bot fetches it.
- Keep the copy. The bot keeps the file and decides every later path from that copy, asking for robots.txt again only when the copy is a day old.
The bot asked for robots.txt before anything else, found its own group, and never sent the request for
/prices/history, because its rules disallow it. Everything this lesson adds is about getting those decisions right.
Checkpoint
Check: What robots.txt is and where it applies
Your product token picks one group
A bot is known in robots.txt by its product token: The short name a bot is known by in robots.txt, made of letters, underscores and hyphens, such as NimbuBot. It is also the first word of the bot's User-Agent header., a short name made of letters,
underscores and hyphens, such as NimbuBot. The token is usually also the first word of the bot’s User-Agent header,
NimbuBot/1.0 (+https://example.org/nimbubot; bots@example.org). RFC 9309 gives three rules for finding the group
(section 2.2.1):
- Match the token against each
User-agentline without regard to case, soNimbuBotandnimbubotare the same bot. - When several groups name the token, combine their rules into one group.
- When no group names the token, use the group for
*; when there is none of that either, no rules apply.
Watch the three rules choose the group for three bots, and one common slip:
The drawing shows the whole scene; its 7 steps are listed below.
Every step as text
- Four groups. The file holds four groups. Each starts with a User-agent line naming a product token, or * for every other bot, and holds its own rules.
- NimbuBot, without case. The token NimbuBot is matched without regard to case, so it finds two groups: NimbuBot and nimbubot.
- Merged into one. RFC 9309 combines every group that names the token. NimbuBot follows 6 rules: the 5 of its own group and Disallow: /drafts/ from the second.
- The star group is gone. A named group replaces the * group entirely. The * rules, such as Disallow: /account/, no longer apply to NimbuBot at all.
- OtherBot falls back. No group names OtherBot, so it follows the * group and its 4 rules, Disallow: /account/ among them.
- ArchiveBot is shut out. ArchiveBot has a group of its own with one rule, Disallow: /. It may fetch nothing but /robots.txt itself.
- The whole header, a slip. Given the full User-Agent instead of the token, urllib.robotparser reads the text before the first /, Mozilla, finds no group and applies *, so NimbuBot's own rules are skipped.
The third rule has a consequence that surprises site owners: a group that names your bot replaces the * group
entirely. NimbuBot’s merged group says nothing about /account/, so NimbuBot may fetch /account/orders although
every other bot is asked to stay out. If a site wants a rule for every bot, it must repeat the rule in each named
group.
Passing the whole User-Agent instead of the token
parser.can_fetch("Mozilla/5.0 (compatible; NimbuBot/1.0; …)", url) asks about a bot called Mozilla. Python’s
urllib.robotparser reads only the text before the first /, finds no such group and applies *, so
/prices/history, which NimbuBot’s group disallows, comes back allowed. Pass the product token, "NimbuBot".
Checkpoint
Check: Choosing the group by product token
The longest matching rule decides
Inside the chosen group, every rule’s path is compared with the address from the first character of its path, with
the query string included, as in the standard’s own examples. Of all the rules that match, the one with the most
characters (octets, in the standard’s words) decides, and when an Allow and a Disallow rule are equally long, Allow wins (section 2.2.2). Three
characters are special: * matches any run of characters, $ at the end of a rule means the address must end there,
and # starts a comment. Before comparing, both sides are percent-encoded the same way, so /%70rices/ is the same
path as /prices/. Two more rules close the gaps: a path that no rule matches is allowed, and /robots.txt itself is
always allowed.
Step through NimbuBot’s group deciding three paths, with the length of every rule that matches:
The drawing shows the whole scene; its 7 steps are listed below.
Every step as text
- The path. NimbuBot wants /prices/today. Its merged group has six rules; each is compared with the path from the first character, in any order.
- Two rules match. Disallow: /prices/ matches, with 8 characters. Allow: /prices/today matches too, with 13. The other four do not match this path.
- The longest wins. The rule with the most characters decides, so Allow: /prices/today wins over the shorter Disallow, although the Disallow comes first in the file.
- A second path. Next, /offers/weekend. Disallow: /offers/ and Allow: /offers/ both match it, with 8 characters each, and nothing longer does.
- A tie goes to Allow. Equally long rules are a tie, and RFC 9309 gives a tie to Allow. NimbuBot may fetch /offers/weekend.
- A wildcard rule. Last, /guides/size-chart.pdf. The * in /*.pdf$ matches any run of characters and the $ requires the path to end in .pdf, so this rule matches and no other does.
- Disallowed. The only matching rule is a Disallow, so the answer is no. With ?print=1 at the end, the $ no longer matches and the same guide is allowed.
The order of the lines in the file never mattered: Disallow: /prices/ comes first, and Allow: /prices/today still
wins because it is longer. The program below puts the twelve questions of the opening to the lesson’s own RFC 9309
matcher, rfc9309.py, and to urllib.robotparser. On Python 3.14.8, the version this track pins, the two agree on
every row:
robots/robots_matrix.py
"""Twelve questions to the practice shop's robots.txt, answered by RFC 9309 and by urllib.robotparser."""
import sys
from urllib.robotparser import RobotFileParser
from nimbu_lab import ROBOTS
from rfc9309 import can_fetch
FULL_UA = "Mozilla/5.0 (compatible; NimbuBot/1.0; +https://bazaar.nimbu.test/bot)"
CASES = [
("NimbuBot", "/prices/today"),
("NimbuBot", "/prices/history"),
("NimbuBot", "/guides/size-chart.pdf"),
("NimbuBot", "/guides/size-chart.pdf?print=1"),
("NimbuBot", "/offers/weekend"),
("NimbuBot", "/drafts/new-range"),
("NimbuBot", "/account/orders"),
("OtherBot", "/account/orders"),
("OtherBot", "/products/?sort=price"),
("ArchiveBot", "/robots.txt"),
("nimbubot", "/%70rices/history"),
(FULL_UA, "/prices/history"),
]
stdlib = RobotFileParser()
stdlib.parse(ROBOTS.splitlines())
print("urllib.robotparser of Python", sys.version.split()[0])
print(f"{'agent':11} {'path':31} RFC 9309 urllib")
for agent, path in CASES:
rfc = "yes" if can_fetch(ROBOTS, agent, path) else "no"
lib = "yes" if stdlib.can_fetch(agent, "https://bazaar.nimbu.test" + path) else "no"
shown = "full UA" if agent == FULL_UA else agent
print(f"{shown:11} {path:31} {rfc:9} {lib:6} {'' if rfc == lib else '<- differs'}")Output
urllib.robotparser of Python 3.14.8 agent path RFC 9309 urllib NimbuBot /prices/today yes yes NimbuBot /prices/history no no NimbuBot /guides/size-chart.pdf no no NimbuBot /guides/size-chart.pdf?print=1 yes yes NimbuBot /offers/weekend yes yes NimbuBot /drafts/new-range no no NimbuBot /account/orders yes yes OtherBot /account/orders no no OtherBot /products/?sort=price no no ArchiveBot /robots.txt yes yes nimbubot /%70rices/history no no full UA /prices/history yes yes
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 robots_matrix.py
rfc9309.py (the matcher) · robots/rfc9309.py
"""RFC 9309 in a few lines: the group for a product token, then the longest matching rule (sections 2.2.1-2.2.3)."""
import re
UNRESERVED = re.compile(r"[A-Za-z0-9._~-]")
def normalise(path):
"""Compare paths percent-encoded: %70 is the letter p, other %xx stay encoded, and non-ASCII is encoded."""
def decode(m):
char = chr(int(m[1], 16))
return char if UNRESERVED.fullmatch(char) else "%" + m[1].upper()
path = re.sub(r"%([0-9A-Fa-f]{2})", decode, path)
return "".join(c if ord(c) < 128 else "".join(f"%{b:02X}" for b in c.encode()) for c in path)
def groups(robots_txt):
"""[(agents, rules, delay)] in file order: a rule is (allow, pattern), delay a Crawl-delay in seconds or None."""
found, agents, rules, delay = [], [], None, None
for line in robots_txt.splitlines():
field, _, value = line.split("#", 1)[0].partition(":")
field, value = field.strip().lower(), value.strip()
if field == "user-agent":
if rules is not None: # a user-agent line after the group's own lines starts a new group
found.append((agents, rules, delay))
agents, rules, delay = [], None, None
agents.append(value.lower())
elif agents and field in ("allow", "disallow", "crawl-delay"): # lines outside a group are ignored
rules = [] if rules is None else rules
if field == "crawl-delay":
delay = float(value) if re.fullmatch(r"\d+(\.\d+)?", value) else delay
elif value: # an empty "Disallow:" allows everything, so it adds no rule
rules.append((field == "allow", normalise(value)))
if agents:
found.append((agents, rules or [], delay))
return found
def group_for(robots_txt, token):
"""The rules and the Crawl-delay for a product token: all its groups merged, or else the * groups."""
found = groups(robots_txt)
chosen = [g for g in found if token.lower() in g[0]] or [g for g in found if "*" in g[0]]
delays = [g[2] for g in chosen if g[2] is not None]
return [rule for g in chosen for rule in g[1]], (delays[0] if delays else None)
def matches(pattern, path):
"""* matches any characters, a final $ ends the match; the match starts at the path's first character."""
regex = "".join(".*" if c == "*" else re.escape(c) for c in pattern.removesuffix("$"))
return re.match(regex + (r"\Z" if pattern.endswith("$") else ""), path) is not None
def can_fetch(robots_txt, token, path):
"""The longest matching rule of the token's group decides; Allow wins a tie; no matching rule allows."""
rules, _ = group_for(robots_txt, token)
path = normalise(path)
if path == "/robots.txt": # always allowed
return True
best = (-1, True)
for allow, pattern in rules:
if matches(pattern, path):
best = max(best, (len(pattern), allow)) # longer wins; at equal length True (Allow) wins
return best[1]nimbu_lab.py (the shop) · robots/nimbu_lab.py
"""Nimbu Bazaar's robots.txt, served inside your program for this lesson. Everything in the shop is invented.
The shop is a WSGI application (PEP 3333). install() points urllib at it, so urllib.robotparser can read the shop's
robots.txt with no network at all; fetch() asks the shop directly, the way the track's own collectors do.
"""
import io
import urllib.error
import urllib.request
import urllib.response
from email.message import Message
from pathlib import Path
from wsgiref.util import setup_testing_defaults
HOST = "bazaar.nimbu.test"
ROBOTS = (Path(__file__).parent / "robots.txt").read_text(encoding="utf-8")
ROBOTS_STATUS = 200 # what the shop answers for /robots.txt: 200, 301, 401, 403, 404, 429, 500 or 503
LOG = [] # every request the shop received: (path, User-Agent)
def app(environ, start_response):
path = environ["PATH_INFO"]
LOG.append((path, environ.get("HTTP_USER_AGENT", "")))
if path == "/robots.txt" and ROBOTS_STATUS == 301:
start_response("301 Moved Permanently", [("Location", f"https://{HOST}/static/robots.txt")])
return [b""]
if path in ("/robots.txt", "/static/robots.txt") and ROBOTS_STATUS in (200, 301):
start_response("200 OK", [("Content-Type", "text/plain; charset=utf-8")])
return [ROBOTS.encode()]
if path == "/robots.txt":
start_response(f"{ROBOTS_STATUS} Error", [("Content-Type", "text/plain")])
return [b"no robots.txt for you right now"]
start_response("200 OK", [("Content-Type", "text/html; charset=utf-8")])
return [b"<p>a page of the shop</p>"]
def fetch(url, user_agent):
"""GET an address of the shop: returns (status, body). Any other host fails like an unknown name."""
host, _, path = url.removeprefix("https://").partition("/")
if host != HOST:
raise urllib.error.URLError(f"cannot resolve {host}")
environ = {"REQUEST_METHOD": "GET", "PATH_INFO": "/" + path, "HTTP_HOST": host, "HTTP_USER_AGENT": user_agent}
setup_testing_defaults(environ)
got = []
body = b"".join(app(environ, lambda status, headers, exc_info=None: got.append(int(status[:3]))))
return got[0], body
class ShopHandler(urllib.request.HTTPSHandler):
"""Answers urllib's https:// requests from app() instead of the network."""
def https_open(self, req):
if req.host != HOST:
raise urllib.error.URLError(f"cannot resolve {req.host}")
environ = {"REQUEST_METHOD": req.get_method(), "PATH_INFO": req.selector, "HTTP_HOST": req.host}
for name, value in req.header_items():
environ["HTTP_" + name.upper().replace("-", "_")] = value
setup_testing_defaults(environ)
answer = {}
def start_response(status, headers, exc_info=None):
answer["status"], answer["headers"] = status, headers
body = b"".join(app(environ, start_response))
headers = Message()
for name, value in answer["headers"]:
headers[name] = value
code, _, reason = answer["status"].partition(" ")
response = urllib.response.addinfourl(io.BytesIO(body), headers, req.full_url, int(code))
response.msg = reason
return response
def install():
"""From now on urllib.request.urlopen(), and so RobotFileParser.read(), asks the shop."""
urllib.request.install_opener(urllib.request.build_opener(ShopHandler()))Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Version note
Press Run and the program runs in the browser runner, whose Python is CPython 3.14.2. Its urllib.robotparser
predates RFC 9309 support: it takes the first group that names the bot and, inside it, the first rule that matches,
reads * and $ as plain characters, and has no special case for /robots.txt. Six of the twelve answers change,
and the matcher’s column stays the same.
In your browser, Pyodide 314.0.7 (CPython 3.14.2) prints:
urllib.robotparser of Python 3.14.2 agent path RFC 9309 urllib NimbuBot /prices/today yes no <- differs NimbuBot /prices/history no no NimbuBot /guides/size-chart.pdf no yes <- differs NimbuBot /guides/size-chart.pdf?print=1 yes yes NimbuBot /offers/weekend yes no <- differs NimbuBot /drafts/new-range no yes <- differs NimbuBot /account/orders yes yes OtherBot /account/orders no no OtherBot /products/?sort=price no yes <- differs ArchiveBot /robots.txt yes no <- differs nimbubot /%70rices/history no no full UA /prices/history yes yes
Read the rows where the browser’s answer changes, and each one names a rule of this section: /prices/today (the
longer Allow wins), the PDF guide ($ and *), /offers/weekend (Allow wins a tie), /drafts/new-range (the
merged nimbubot group), the sorted product list (* in the middle of a rule) and ArchiveBot’s /robots.txt. The
last row is the trap of the previous section in numbers: with the whole User-Agent, both parsers apply the * group
and allow a path that NimbuBot’s own group forbids.
Key idea
Pick the group by the product token alone, merge every group that names it, and let the longest matching rule
decide, with Allow on a tie: the order of the lines never changes the answer.
Checkpoint
Check: The longest matching rule
The answer to the robots.txt request comes first
Before any rule applies, the request for robots.txt itself gets an answer, and RFC 9309 says what each kind of answer means (section 2.3.1). A collector needs a decision for every one of them, including the ones the standard leaves open:
| Answer to the request | RFC 9309 | A careful collector |
|---|---|---|
2xx with the file |
Follow the rules it can parse | Follows the rules |
3xx redirect |
Follow at least five redirects in a row | Follows up to five, then treats the file as unavailable |
401, 403 |
Unavailable: the crawler may access the site | Disallows everything: the whole site likely needs an account |
404 and other 4xx |
Unavailable: the crawler may access the site | Applies no rules, and crawls politely |
429 |
A 4xx, so unavailable by the letter |
Disallows everything for now, and asks again later |
5xx, network error, timeout |
Unreachable: assume complete disallow | Disallows everything, and asks again later |
The two stricter choices in the right column are allowed by the standard, which says only that a crawler may go
on when the file is unavailable. A 429 is the site saying the client is sending too much, and Google’s crawlers
also leave 429 out of the rule that treats a 4xx as no restrictions
(Google’s robots.txt documentation).
Python’s parser makes its own choices. The next program serves the shop’s robots.txt with each of those answers in
turn, lets RobotFileParser.read() fetch it through urllib, and prints what the parser concludes:
robots/robots_status.py
"""What urllib.robotparser concludes from each answer to its robots.txt request, and what it sends."""
from urllib.error import URLError
from urllib.robotparser import RobotFileParser
import nimbu_lab
nimbu_lab.install() # urllib's requests now go to the practice shop, inside this program
SHOP = "https://bazaar.nimbu.test"
def verdict(parser):
"""Two questions for NimbuBot: its group allows /products/ and disallows /prices/history."""
products = parser.can_fetch("NimbuBot", SHOP + "/products/")
history = parser.can_fetch("NimbuBot", SHOP + "/prices/history")
if products and not history:
return "follows the rules: /products/ yes, /prices/history no"
return "allows everything" if products else "disallows everything"
print(f"{'answer':7} what urllib.robotparser concludes")
for status in [200, 301, 401, 403, 404, 429, 500, 503]:
nimbu_lab.ROBOTS_STATUS = status
parser = RobotFileParser(SHOP + "/robots.txt")
parser.read()
print(f"{status:<7} {verdict(parser)}")
parser = RobotFileParser("https://nimbu-mirror.test/robots.txt") # a host that cannot be reached
try:
parser.read()
except URLError as err:
print(f"{'none':7} read() raised URLError ({err.reason}); can_fetch says {parser.can_fetch('NimbuBot', '/')}")
print("User-Agent that read() sent:", nimbu_lab.LOG[0][1])Output
answer what urllib.robotparser concludes 200 follows the rules: /products/ yes, /prices/history no 301 follows the rules: /products/ yes, /prices/history no 401 disallows everything 403 disallows everything 404 allows everything 429 allows everything 500 disallows everything 503 disallows everything none read() raised URLError (cannot resolve nimbu-mirror.test); can_fetch says False User-Agent that read() sent: Python-urllib/3.14
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 robots_status.py
nimbu_lab.py (the shop) · robots/nimbu_lab.py
"""Nimbu Bazaar's robots.txt, served inside your program for this lesson. Everything in the shop is invented.
The shop is a WSGI application (PEP 3333). install() points urllib at it, so urllib.robotparser can read the shop's
robots.txt with no network at all; fetch() asks the shop directly, the way the track's own collectors do.
"""
import io
import urllib.error
import urllib.request
import urllib.response
from email.message import Message
from pathlib import Path
from wsgiref.util import setup_testing_defaults
HOST = "bazaar.nimbu.test"
ROBOTS = (Path(__file__).parent / "robots.txt").read_text(encoding="utf-8")
ROBOTS_STATUS = 200 # what the shop answers for /robots.txt: 200, 301, 401, 403, 404, 429, 500 or 503
LOG = [] # every request the shop received: (path, User-Agent)
def app(environ, start_response):
path = environ["PATH_INFO"]
LOG.append((path, environ.get("HTTP_USER_AGENT", "")))
if path == "/robots.txt" and ROBOTS_STATUS == 301:
start_response("301 Moved Permanently", [("Location", f"https://{HOST}/static/robots.txt")])
return [b""]
if path in ("/robots.txt", "/static/robots.txt") and ROBOTS_STATUS in (200, 301):
start_response("200 OK", [("Content-Type", "text/plain; charset=utf-8")])
return [ROBOTS.encode()]
if path == "/robots.txt":
start_response(f"{ROBOTS_STATUS} Error", [("Content-Type", "text/plain")])
return [b"no robots.txt for you right now"]
start_response("200 OK", [("Content-Type", "text/html; charset=utf-8")])
return [b"<p>a page of the shop</p>"]
def fetch(url, user_agent):
"""GET an address of the shop: returns (status, body). Any other host fails like an unknown name."""
host, _, path = url.removeprefix("https://").partition("/")
if host != HOST:
raise urllib.error.URLError(f"cannot resolve {host}")
environ = {"REQUEST_METHOD": "GET", "PATH_INFO": "/" + path, "HTTP_HOST": host, "HTTP_USER_AGENT": user_agent}
setup_testing_defaults(environ)
got = []
body = b"".join(app(environ, lambda status, headers, exc_info=None: got.append(int(status[:3]))))
return got[0], body
class ShopHandler(urllib.request.HTTPSHandler):
"""Answers urllib's https:// requests from app() instead of the network."""
def https_open(self, req):
if req.host != HOST:
raise urllib.error.URLError(f"cannot resolve {req.host}")
environ = {"REQUEST_METHOD": req.get_method(), "PATH_INFO": req.selector, "HTTP_HOST": req.host}
for name, value in req.header_items():
environ["HTTP_" + name.upper().replace("-", "_")] = value
setup_testing_defaults(environ)
answer = {}
def start_response(status, headers, exc_info=None):
answer["status"], answer["headers"] = status, headers
body = b"".join(app(environ, start_response))
headers = Message()
for name, value in answer["headers"]:
headers[name] = value
code, _, reason = answer["status"].partition(" ")
response = urllib.response.addinfourl(io.BytesIO(body), headers, req.full_url, int(code))
response.msg = reason
return response
def install():
"""From now on urllib.request.urlopen(), and so RobotFileParser.read(), asks the shop."""
urllib.request.install_opener(urllib.request.build_opener(ShopHandler()))Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The parser disallows everything for 401 and 403, and for 5xx (since 3.14.7 it sets that explicitly, as RFC 9309
asks). It allows everything for 404 and also for 429, which a careful collector should not. A host that cannot be
reached makes read() raise URLError, so the program must catch it. And the last line shows what read() sent as
its User-Agent: Python-urllib/3.14, a library default that Wikimedia’s User-Agent policy names among the generic
strings it discourages. read() also takes no timeout, so a server that never answers keeps it waiting.
Fetch robots.txt yourself, then parse it
Request the file with your own client, with a timeout and an honest User-Agent, decide from the status as in the
table, and give the text to RobotFileParser.parse(). The parser then only does what it is good at: matching rules.
Checkpoint
Check: Answers to the robots.txt request
One copy is good for a day
A collector reads robots.txt again from time to time, because sites change their rules, and RFC 9309 sets the limit: a crawler should not use a cached copy for more than 24 hours, unless the file is unreachable (section 2.4). A crawler may also follow the file’s standard HTTP cache headers, and Google’s crawlers, for example, generally keep the file for up to 24 hours. The same standard asks a parser to read at least 500 KiB of the file and allows it to stop there (section 2.5).
The cache below keeps one copy per site, refreshes it after 24 hours, and keeps the old copy while a refresh fails.
The shop changes as the program runs: it answers 503 at hour 24.5, and an hour later serves a new robots.txt
without the line Allow: /prices/today. Step through the five questions first:
The drawing shows the whole scene; its 7 steps are listed below.
Every step as text
- Hour 0. The collector asks whether it may fetch /prices/today. The cache has no copy for the shop yet, so it asks for robots.txt.
- Kept until hour 24. The shop answers 200. The cache keeps the file until hour 24, and the answer for /prices/today is yes.
- Hours 2 and 23.5. Two more questions, at hour 2 and hour 23.5, are answered from the copy. No request reaches the shop: /prices/history is no, /prices/today is yes.
- Hour 24.5, refused. The copy is now older than a day, so the cache asks again. The shop answers 503, a server error: the file is unreachable.
- The old copy stays. While the file is unreachable, RFC 9309 lets a crawler keep its cached copy past 24 hours. The cache keeps the old rules and plans to ask again in an hour.
- Hour 25.5, new rules. The shop is back with a new file that no longer allows /prices/today. The cache replaces its copy, and the answer for /prices/today turns to no.
- Three requests. Five questions over 25.5 hours cost the shop 3 requests for robots.txt. A collector that asked for the file before every page would have sent 5.
The program asks the same five questions with a clock it controls, and counts the shop’s robots.txt requests:
robots/robots_cache.py
"""One copy of robots.txt per site, kept at most 24 hours, and longer only while the file cannot be reached."""
from urllib.error import URLError
import nimbu_lab
from rfc9309 import can_fetch
USER_AGENT = "NimbuBot/1.0 (+https://example.org/nimbubot; bots@example.org)"
TOKEN = "NimbuBot"
HOUR, DAY = 3600, 24 * 3600
LIMIT = 500 * 1024 # RFC 9309 asks parsers to read at least 500 KiB; this cache reads that much and no more
class RobotsCache:
def __init__(self, clock):
self.clock = clock # a function that returns the time in seconds
self.copies = {} # site -> {"rules": robots.txt text, "until": when to fetch it again}
def rules(self, site):
now, copy = self.clock(), self.copies.get(site)
if copy and now < copy["until"]:
return copy["rules"], "kept copy"
try:
status, body = nimbu_lab.fetch(f"https://{site}/robots.txt", USER_AGENT)
except URLError:
status, body = None, b""
if status is not None and 200 <= status < 300:
text = body[:LIMIT].decode("utf-8", "replace")
self.copies[site] = {"rules": text, "until": now + DAY}
return text, f"fetched: {status}"
if status is not None and 400 <= status < 500 and status not in (401, 403, 429):
self.copies[site] = {"rules": "", "until": now + DAY} # unavailable: no rules apply
return "", f"fetched: {status}, no rules"
if copy: # unreachable, 429, 401 or 403: keep the old copy and ask again in an hour
copy["until"] = now + HOUR
return copy["rules"], f"fetched: {status}, old copy kept"
self.copies[site] = {"rules": "User-agent: *\nDisallow: /", "until": now + HOUR}
return self.copies[site]["rules"], f"fetched: {status}, nothing allowed yet"
hours = [0]
cache = RobotsCache(clock=lambda: hours[0] * HOUR)
def ask(at, path, change=None):
hours[0] = at
if change:
change()
text, how = cache.rules(nimbu_lab.HOST)
answer = "yes" if can_fetch(text, TOKEN, path) else "no"
print(f"hour {at:4} {path:16} {answer:4} {how}")
def shop_down():
nimbu_lab.ROBOTS_STATUS = 503
def shop_back_with_new_rules():
nimbu_lab.ROBOTS_STATUS = 200
nimbu_lab.ROBOTS = nimbu_lab.ROBOTS.replace("Allow: /prices/today\n", "")
ask(0, "/prices/today")
ask(2, "/prices/history")
ask(23.5, "/prices/today")
ask(24.5, "/prices/today", change=shop_down)
ask(25.5, "/prices/today", change=shop_back_with_new_rules)
print("robots.txt requests the shop received:", sum(path == "/robots.txt" for path, _ in nimbu_lab.LOG))Output
hour 0 /prices/today yes fetched: 200 hour 2 /prices/history no kept copy hour 23.5 /prices/today yes kept copy hour 24.5 /prices/today yes fetched: 503, old copy kept hour 25.5 /prices/today no fetched: 200 robots.txt requests the shop received: 3
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 robots_cache.py
rfc9309.py (the matcher) · robots/rfc9309.py
"""RFC 9309 in a few lines: the group for a product token, then the longest matching rule (sections 2.2.1-2.2.3)."""
import re
UNRESERVED = re.compile(r"[A-Za-z0-9._~-]")
def normalise(path):
"""Compare paths percent-encoded: %70 is the letter p, other %xx stay encoded, and non-ASCII is encoded."""
def decode(m):
char = chr(int(m[1], 16))
return char if UNRESERVED.fullmatch(char) else "%" + m[1].upper()
path = re.sub(r"%([0-9A-Fa-f]{2})", decode, path)
return "".join(c if ord(c) < 128 else "".join(f"%{b:02X}" for b in c.encode()) for c in path)
def groups(robots_txt):
"""[(agents, rules, delay)] in file order: a rule is (allow, pattern), delay a Crawl-delay in seconds or None."""
found, agents, rules, delay = [], [], None, None
for line in robots_txt.splitlines():
field, _, value = line.split("#", 1)[0].partition(":")
field, value = field.strip().lower(), value.strip()
if field == "user-agent":
if rules is not None: # a user-agent line after the group's own lines starts a new group
found.append((agents, rules, delay))
agents, rules, delay = [], None, None
agents.append(value.lower())
elif agents and field in ("allow", "disallow", "crawl-delay"): # lines outside a group are ignored
rules = [] if rules is None else rules
if field == "crawl-delay":
delay = float(value) if re.fullmatch(r"\d+(\.\d+)?", value) else delay
elif value: # an empty "Disallow:" allows everything, so it adds no rule
rules.append((field == "allow", normalise(value)))
if agents:
found.append((agents, rules or [], delay))
return found
def group_for(robots_txt, token):
"""The rules and the Crawl-delay for a product token: all its groups merged, or else the * groups."""
found = groups(robots_txt)
chosen = [g for g in found if token.lower() in g[0]] or [g for g in found if "*" in g[0]]
delays = [g[2] for g in chosen if g[2] is not None]
return [rule for g in chosen for rule in g[1]], (delays[0] if delays else None)
def matches(pattern, path):
"""* matches any characters, a final $ ends the match; the match starts at the path's first character."""
regex = "".join(".*" if c == "*" else re.escape(c) for c in pattern.removesuffix("$"))
return re.match(regex + (r"\Z" if pattern.endswith("$") else ""), path) is not None
def can_fetch(robots_txt, token, path):
"""The longest matching rule of the token's group decides; Allow wins a tie; no matching rule allows."""
rules, _ = group_for(robots_txt, token)
path = normalise(path)
if path == "/robots.txt": # always allowed
return True
best = (-1, True)
for allow, pattern in rules:
if matches(pattern, path):
best = max(best, (len(pattern), allow)) # longer wins; at equal length True (Allow) wins
return best[1]nimbu_lab.py (the shop) · robots/nimbu_lab.py
"""Nimbu Bazaar's robots.txt, served inside your program for this lesson. Everything in the shop is invented.
The shop is a WSGI application (PEP 3333). install() points urllib at it, so urllib.robotparser can read the shop's
robots.txt with no network at all; fetch() asks the shop directly, the way the track's own collectors do.
"""
import io
import urllib.error
import urllib.request
import urllib.response
from email.message import Message
from pathlib import Path
from wsgiref.util import setup_testing_defaults
HOST = "bazaar.nimbu.test"
ROBOTS = (Path(__file__).parent / "robots.txt").read_text(encoding="utf-8")
ROBOTS_STATUS = 200 # what the shop answers for /robots.txt: 200, 301, 401, 403, 404, 429, 500 or 503
LOG = [] # every request the shop received: (path, User-Agent)
def app(environ, start_response):
path = environ["PATH_INFO"]
LOG.append((path, environ.get("HTTP_USER_AGENT", "")))
if path == "/robots.txt" and ROBOTS_STATUS == 301:
start_response("301 Moved Permanently", [("Location", f"https://{HOST}/static/robots.txt")])
return [b""]
if path in ("/robots.txt", "/static/robots.txt") and ROBOTS_STATUS in (200, 301):
start_response("200 OK", [("Content-Type", "text/plain; charset=utf-8")])
return [ROBOTS.encode()]
if path == "/robots.txt":
start_response(f"{ROBOTS_STATUS} Error", [("Content-Type", "text/plain")])
return [b"no robots.txt for you right now"]
start_response("200 OK", [("Content-Type", "text/html; charset=utf-8")])
return [b"<p>a page of the shop</p>"]
def fetch(url, user_agent):
"""GET an address of the shop: returns (status, body). Any other host fails like an unknown name."""
host, _, path = url.removeprefix("https://").partition("/")
if host != HOST:
raise urllib.error.URLError(f"cannot resolve {host}")
environ = {"REQUEST_METHOD": "GET", "PATH_INFO": "/" + path, "HTTP_HOST": host, "HTTP_USER_AGENT": user_agent}
setup_testing_defaults(environ)
got = []
body = b"".join(app(environ, lambda status, headers, exc_info=None: got.append(int(status[:3]))))
return got[0], body
class ShopHandler(urllib.request.HTTPSHandler):
"""Answers urllib's https:// requests from app() instead of the network."""
def https_open(self, req):
if req.host != HOST:
raise urllib.error.URLError(f"cannot resolve {req.host}")
environ = {"REQUEST_METHOD": req.get_method(), "PATH_INFO": req.selector, "HTTP_HOST": req.host}
for name, value in req.header_items():
environ["HTTP_" + name.upper().replace("-", "_")] = value
setup_testing_defaults(environ)
answer = {}
def start_response(status, headers, exc_info=None):
answer["status"], answer["headers"] = status, headers
body = b"".join(app(environ, start_response))
headers = Message()
for name, value in answer["headers"]:
headers[name] = value
code, _, reason = answer["status"].partition(" ")
response = urllib.response.addinfourl(io.BytesIO(body), headers, req.full_url, int(code))
response.msg = reason
return response
def install():
"""From now on urllib.request.urlopen(), and so RobotFileParser.read(), asks the shop."""
urllib.request.install_opener(urllib.request.build_opener(ShopHandler()))Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Twenty-five and a half hours of questions cost the shop 3 requests for robots.txt. At hour 24.5 the copy had expired
and the refresh failed with 503, so the cache kept the old rules and set itself to ask again an hour later, as the
standard allows while the file is unreachable. At hour 25.5 the new file arrived, and /prices/today, allowed all day,
became disallowed. If a file stays unreachable for a long time, 30 days in the standard’s example, a crawler may
treat it as unavailable or keep using its cached copy (section 2.3.1.4).
Move the clock
Press Edit and change ask(23.5, …) to ask(24, …). The copy from hour 0 is a day old at hour 24, so the cache
fetches a new one, fresh until hour 48. The 503 and the new rules then go unseen: /prices/today stays allowed at
hour 25.5, and the shop counts 2 requests. A copy spares the shop, and new rules can take up to a day to arrive.
Checkpoint
Check: Caching robots.txt for a day
Crawl-delay is outside the standard
A robots.txt can carry a Crawl-delay line, the number of seconds a site asks a bot to wait between requests.
RFC 9309 does not define it, and Google’s crawlers do not support it. A polite collector honours it anyway, because
it is the site’s wish written down, but it must read it correctly first:
robots/crawl_delay.py
"""Crawl-delay is not part of RFC 9309. What urllib.robotparser reads from it, and the pace a polite bot picks."""
from urllib.robotparser import RobotFileParser
from nimbu_lab import ROBOTS
from rfc9309 import group_for
stdlib = RobotFileParser()
stdlib.parse(ROBOTS.splitlines())
for token in ["NimbuBot", "OtherBot"]:
_, written = group_for(ROBOTS, token)
print(f"{token:9} urllib.robotparser: {stdlib.crawl_delay(token)!s:5} as written: {written}")
TERMS_GAP = 1.0 # the shop's terms ask for at most one request a second
_, delay = group_for(ROBOTS, "NimbuBot")
pace = max(TERMS_GAP, delay or 0)
print(f"NimbuBot waits {pace} s between requests: 10 listing pages take {10 * pace:.0f} s")Output
NimbuBot urllib.robotparser: None as written: 2.5 OtherBot urllib.robotparser: 2 as written: 2.0 NimbuBot waits 2.5 s between requests: 10 listing pages take 25 s
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 crawl_delay.py
rfc9309.py (the matcher) · robots/rfc9309.py
"""RFC 9309 in a few lines: the group for a product token, then the longest matching rule (sections 2.2.1-2.2.3)."""
import re
UNRESERVED = re.compile(r"[A-Za-z0-9._~-]")
def normalise(path):
"""Compare paths percent-encoded: %70 is the letter p, other %xx stay encoded, and non-ASCII is encoded."""
def decode(m):
char = chr(int(m[1], 16))
return char if UNRESERVED.fullmatch(char) else "%" + m[1].upper()
path = re.sub(r"%([0-9A-Fa-f]{2})", decode, path)
return "".join(c if ord(c) < 128 else "".join(f"%{b:02X}" for b in c.encode()) for c in path)
def groups(robots_txt):
"""[(agents, rules, delay)] in file order: a rule is (allow, pattern), delay a Crawl-delay in seconds or None."""
found, agents, rules, delay = [], [], None, None
for line in robots_txt.splitlines():
field, _, value = line.split("#", 1)[0].partition(":")
field, value = field.strip().lower(), value.strip()
if field == "user-agent":
if rules is not None: # a user-agent line after the group's own lines starts a new group
found.append((agents, rules, delay))
agents, rules, delay = [], None, None
agents.append(value.lower())
elif agents and field in ("allow", "disallow", "crawl-delay"): # lines outside a group are ignored
rules = [] if rules is None else rules
if field == "crawl-delay":
delay = float(value) if re.fullmatch(r"\d+(\.\d+)?", value) else delay
elif value: # an empty "Disallow:" allows everything, so it adds no rule
rules.append((field == "allow", normalise(value)))
if agents:
found.append((agents, rules or [], delay))
return found
def group_for(robots_txt, token):
"""The rules and the Crawl-delay for a product token: all its groups merged, or else the * groups."""
found = groups(robots_txt)
chosen = [g for g in found if token.lower() in g[0]] or [g for g in found if "*" in g[0]]
delays = [g[2] for g in chosen if g[2] is not None]
return [rule for g in chosen for rule in g[1]], (delays[0] if delays else None)
def matches(pattern, path):
"""* matches any characters, a final $ ends the match; the match starts at the path's first character."""
regex = "".join(".*" if c == "*" else re.escape(c) for c in pattern.removesuffix("$"))
return re.match(regex + (r"\Z" if pattern.endswith("$") else ""), path) is not None
def can_fetch(robots_txt, token, path):
"""The longest matching rule of the token's group decides; Allow wins a tie; no matching rule allows."""
rules, _ = group_for(robots_txt, token)
path = normalise(path)
if path == "/robots.txt": # always allowed
return True
best = (-1, True)
for allow, pattern in rules:
if matches(pattern, path):
best = max(best, (len(pattern), allow)) # longer wins; at equal length True (Allow) wins
return best[1]nimbu_lab.py (the shop) · robots/nimbu_lab.py
"""Nimbu Bazaar's robots.txt, served inside your program for this lesson. Everything in the shop is invented.
The shop is a WSGI application (PEP 3333). install() points urllib at it, so urllib.robotparser can read the shop's
robots.txt with no network at all; fetch() asks the shop directly, the way the track's own collectors do.
"""
import io
import urllib.error
import urllib.request
import urllib.response
from email.message import Message
from pathlib import Path
from wsgiref.util import setup_testing_defaults
HOST = "bazaar.nimbu.test"
ROBOTS = (Path(__file__).parent / "robots.txt").read_text(encoding="utf-8")
ROBOTS_STATUS = 200 # what the shop answers for /robots.txt: 200, 301, 401, 403, 404, 429, 500 or 503
LOG = [] # every request the shop received: (path, User-Agent)
def app(environ, start_response):
path = environ["PATH_INFO"]
LOG.append((path, environ.get("HTTP_USER_AGENT", "")))
if path == "/robots.txt" and ROBOTS_STATUS == 301:
start_response("301 Moved Permanently", [("Location", f"https://{HOST}/static/robots.txt")])
return [b""]
if path in ("/robots.txt", "/static/robots.txt") and ROBOTS_STATUS in (200, 301):
start_response("200 OK", [("Content-Type", "text/plain; charset=utf-8")])
return [ROBOTS.encode()]
if path == "/robots.txt":
start_response(f"{ROBOTS_STATUS} Error", [("Content-Type", "text/plain")])
return [b"no robots.txt for you right now"]
start_response("200 OK", [("Content-Type", "text/html; charset=utf-8")])
return [b"<p>a page of the shop</p>"]
def fetch(url, user_agent):
"""GET an address of the shop: returns (status, body). Any other host fails like an unknown name."""
host, _, path = url.removeprefix("https://").partition("/")
if host != HOST:
raise urllib.error.URLError(f"cannot resolve {host}")
environ = {"REQUEST_METHOD": "GET", "PATH_INFO": "/" + path, "HTTP_HOST": host, "HTTP_USER_AGENT": user_agent}
setup_testing_defaults(environ)
got = []
body = b"".join(app(environ, lambda status, headers, exc_info=None: got.append(int(status[:3]))))
return got[0], body
class ShopHandler(urllib.request.HTTPSHandler):
"""Answers urllib's https:// requests from app() instead of the network."""
def https_open(self, req):
if req.host != HOST:
raise urllib.error.URLError(f"cannot resolve {req.host}")
environ = {"REQUEST_METHOD": req.get_method(), "PATH_INFO": req.selector, "HTTP_HOST": req.host}
for name, value in req.header_items():
environ["HTTP_" + name.upper().replace("-", "_")] = value
setup_testing_defaults(environ)
answer = {}
def start_response(status, headers, exc_info=None):
answer["status"], answer["headers"] = status, headers
body = b"".join(app(environ, start_response))
headers = Message()
for name, value in answer["headers"]:
headers[name] = value
code, _, reason = answer["status"].partition(" ")
response = urllib.response.addinfourl(io.BytesIO(body), headers, req.full_url, int(code))
response.msg = reason
return response
def install():
"""From now on urllib.request.urlopen(), and so RobotFileParser.read(), asks the shop."""
urllib.request.install_opener(urllib.request.build_opener(ShopHandler()))Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
NimbuBot’s group asks for Crawl-delay: 2.5, and urllib.robotparser returns None: it accepts whole numbers only,
so a delay with decimals vanishes without a warning, and a collector that trusts it would not wait at all. The *
group’s 2 comes through. Protego, a robots.txt parser for Python, returns the value as a float, as the matcher here
does (Protego). The program then takes the larger of the delay and the shop’s
terms, which ask for one request a second at most, so NimbuBot waits 2.5 seconds and reads the 10 listing pages in
25 seconds.
Checkpoint
Check: Crawl-delay
Choose a parser whose answers you have checked
The twelve questions showed that one module of one language gave different answers in two patch releases. The
history of urllib.robotparser in Python 3.14 explains where those answers came from:
- 3.14.2
The Python of the browser runner first match wins
Pyodide 314.0.7 runs CPython 3.14.2. Its parser takes the first group that names the bot and the first rule that matches, reads * and $ inside a rule as plain characters, and has no special case for /robots.txt: six of the twelve answers differ from RFC 9309.
- 3.14.5
The rules of RFC 9309 longest match
The module now follows RFC 9309: groups for one token are merged, the longest match decides, Allow wins a tie, * and $ are wildcards, and /robots.txt is always allowed. A wildcard rule is measured by the part of the path it matched.
- 3.14.7
Unreachable means disallow server errors
A 5xx answer to read() now sets "disallow everything" explicitly, as the standard asks. A host that cannot be reached still makes read() raise URLError, and Crawl-delay still has to be a whole number.
- 3.14.8
The version this track pins the pin
On 3.14.8 the parser agrees with the lesson's own matcher on all twelve questions. It still sends Python-urllib/3.14 as its User-Agent and drops a Crawl-delay with decimals.
The timeline has 4 events, from 3.14.2 to 3.14.8.
Outside the standard library, two parsers are common. Protego is a pure-Python parser, under the BSD licence, that implements the matching rules of RFC 9309 and leaves fetching the file to you. robots-parser does the same for Node.js and names RFC 9309 as its target. Whichever you choose, run the twelve questions of this lesson through it before you trust it: a parser is only as good as the cases you have checked it on.
Twelve questions are still only twelve. Give both readers of this lesson a group with Allow: /guides/ and
Disallow: /*.pdf$, and the size-chart guide splits them on Python 3.14.8:
"""Two readings of "the longest match": urllib.robotparser measures a wildcard rule by the part of the path it
matched, while the lesson's RFC 9309 matcher counts the characters of the rule itself."""
import sys
from urllib.robotparser import RobotFileParser
from rfc9309 import can_fetch
ROBOTS = "User-agent: *\nAllow: /guides/\nDisallow: /*.pdf$\n"
stdlib = RobotFileParser()
stdlib.parse(ROBOTS.splitlines())
print("urllib.robotparser of Python", sys.version.split()[0])
print(f"{'path':24} matcher urllib")
for path in ["/guides/size-chart.pdf", "/guides/sizes", "/prices.pdf"]:
rfc = "yes" if can_fetch(ROBOTS, "NimbuBot", path) else "no"
lib = "yes" if stdlib.can_fetch("NimbuBot", "https://bazaar.nimbu.test" + path) else "no"
print(f"{path:24} {rfc:8} {lib}")Output
urllib.robotparser of Python 3.14.8 path matcher urllib /guides/size-chart.pdf yes no /guides/sizes yes yes /prices.pdf no no
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 wildcard_length.py
urllib.robotparser measures a wildcard rule by the part of the path it matched, all 22 characters of
/guides/size-chart.pdf, so the Disallow beats the 8 characters of /guides/. The lesson’s matcher counts the 7
characters of the rule itself, and the Allow wins; on the other two paths they agree. RFC 9309 asks only for the
most specific match, the one with the most octets, and the two parsers read that differently, so test the rules your
own sites use in every parser you rely on.
Arguments in the wrong order
urllib.robotparser asks can_fetch(useragent, url), Protego asks can_fetch(url, user_agent), and robots-parser
asks isAllowed(url, ua). Swap them and the parser looks for a bot named after your address and falls back to the
* group, with no error to warn you. Check the order in each library’s documentation.
Checkpoint
Check: Choosing a parser
Practise: build the rules a crawler follows
Two steps build the robots.txt logic of a collector. The first finds the rule that decides a path for a bot, with the practice shop’s file as its test data; the second turns the answer to the robots.txt request into what the bot may do. Both steps run their sample tests in this page and open with a hint you can read before writing a line:
Code lab · 2 steps
Build a robots.txt decision for a crawler
Find the rule that decides a path for a bot, then decide what to do with each answer to the robots.txt request.
Exercise · Medium · Python
Find the robots.txt rule that decides a path
deciding.py already reads a robots.txt into groups with parse(robots_txt), a list of {"agents": [...], "rules": [(kind, pattern), ...]} in file order, with every agent in lower case and every rule as written, such as ("Disallow", "/prices/"). matches(pattern, path) tells you whether a rule's pattern matches a path, with * and a final $ as wildcards. Paths in this exercise are already percent-encoded alike.
- Write
choose_rules(groups, token): the rules of every group that names the product token, in any case, merged in file order; when no group names it, the rules of every*group; when there is none of those either,[]. - Write
deciding_rule(robots_txt, token, path): the rule that decides the path for the token, written as"Allow: /prices/today"or"Disallow: /prices/", orNonewhen no rule matches (the path is then allowed). The longest pattern that matches decides, anAllowwins over an equally longDisallow, and/robots.txtis always allowed, so it returnsNonefor it.
The sample tests use the practice shop's robots.txt from the lesson and say which rule of RFC 9309 a wrong answer breaks.
Starter code · deciding.py
import re
def parse(robots_txt):
"""The groups of a robots.txt in file order: {"agents": [lower-case tokens], "rules": [(kind, pattern)]}."""
groups, current = [], None
for line in robots_txt.splitlines():
field, _, value = line.split("#", 1)[0].partition(":")
field, value = field.strip().lower(), value.strip()
if field == "user-agent":
if current is None or current["rules"]:
current = {"agents": [], "rules": []}
groups.append(current)
current["agents"].append(value.lower())
elif field in ("allow", "disallow") and current is not None and value:
current["rules"].append((field.capitalize(), value))
return groups
def matches(pattern, path):
"""True when the pattern matches the path from its first character: * is any run, a final $ the end."""
regex = "".join(".*" if c == "*" else re.escape(c) for c in pattern.removesuffix("$"))
return re.match(regex + (r"\Z" if pattern.endswith("$") else ""), path) is not None
def choose_rules(groups, token):
"""The rules of every group that names the token (any case), else of the * groups, else []."""
# Replace this: the first group is rarely the right one.
return groups[0]["rules"] if groups else []
def deciding_rule(robots_txt, token, path):
"""The rule that decides the path for the token, as "Allow: …" or "Disallow: …", or None (allowed)."""
# Replace this: the first matching rule is not always the deciding one.
for kind, pattern in choose_rules(parse(robots_txt), token):
if matches(pattern, path):
return f"{kind}: {pattern}"
return NoneThe sample tests · test_deciding.py
from deciding import choose_rules, deciding_rule, parse
ROBOTS = """\
User-agent: *
Disallow: /account/
Disallow: /people/
Disallow: /products/*?sort=
Allow: /products/
User-agent: NimbuBot
Disallow: /prices/
Allow: /prices/today
Disallow: /*.pdf$
Disallow: /offers/
Allow: /offers/
User-agent: nimbubot
Disallow: /drafts/
User-agent: ArchiveBot
Disallow: /
"""
def check(token, path, want, why):
got = deciding_rule(ROBOTS, token, path)
assert got == want, f"{token} {path}: expected {want!r} ({why}), got {got!r}"
def test_merges_every_group_that_names_the_token():
"""NimbuBot and nimbubot are one token, so their rules merge in file order"""
rules = choose_rules(parse(ROBOTS), "NimbuBot")
assert ("Disallow", "/drafts/") in rules, "the nimbubot group names the same token in another case: merge it"
assert rules[0] == ("Disallow", "/prices/") and len(rules) == 6, "NimbuBot follows the 5 rules of its group and the 1 of nimbubot"
assert ("Disallow", "/account/") not in rules, "a group that names the bot replaces the * group"
def test_falls_back_to_star_only_when_no_group_names_the_token():
"""OtherBot gets the * rules; with no * group a stranger gets none"""
assert choose_rules(parse(ROBOTS), "OtherBot") == [("Disallow", "/account/"), ("Disallow", "/people/"), ("Disallow", "/products/*?sort="), ("Allow", "/products/")]
assert choose_rules(parse("User-agent: BotA\nDisallow: /\n"), "BotC") == [], "no matching group and no * group: no rules apply"
def test_the_longest_rule_decides():
"""a longer Allow beats a shorter Disallow that comes first"""
check("NimbuBot", "/prices/today", "Allow: /prices/today", "13 characters beat 8")
check("NimbuBot", "/prices/history", "Disallow: /prices/", "only that rule matches")
check("OtherBot", "/products/?sort=price", "Disallow: /products/*?sort=", "the wildcard rule is the longer match")
def test_allow_wins_a_tie():
"""equally long Allow and Disallow rules: Allow"""
check("NimbuBot", "/offers/weekend", "Allow: /offers/", "both are 8 characters, and a tie goes to Allow")
def test_wildcards_and_the_end_anchor():
"""* matches any run, a final $ the end of the path"""
check("NimbuBot", "/guides/size-chart.pdf", "Disallow: /*.pdf$", "the path ends in .pdf")
check("NimbuBot", "/guides/size-chart.pdf?print=1", None, "with a query after .pdf, the $ no longer matches")
def test_no_matching_rule_and_robots_txt():
"""no matching rule allows; /robots.txt is always allowed"""
check("NimbuBot", "/account/orders", None, "NimbuBot's group says nothing about /account/")
check("NimbuBot", "/drafts/new-range", "Disallow: /drafts/", "the merged nimbubot group")
check("ArchiveBot", "/robots.txt", None, "/robots.txt is allowed even under Disallow: /")
check("ArchiveBot", "/products/", "Disallow: /", "ArchiveBot's only rule")
check("archivebot", "/news/", "Disallow: /", "tokens match in any case")A hint
For choose_rules, first collect the groups whose agents contain token.lower(); only if that list is empty, take the groups that contain "*". For deciding_rule, compare each matching rule by (len(pattern), kind == "Allow") and keep the largest: a longer pattern wins, and at equal length True beats False, so Allow wins the tie.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
Exercise · Easy · Python
Turn the robots.txt answer into a decision
Write the two decisions a collector makes before it reads a single rule.
policy(status, redirects=0)takes the final status of the request for robots.txt, orNonewhen the request failed with a network error or a timeout, and the number of redirects followed on the way. It returns one of three strings, as the lesson's table does:"follow the rules"for a2xxanswer reached in at most 5 redirects;"no rules"for a4xxanswer other than401,403and429, and for more than 5 redirects (the file is then unavailable);"disallow everything"for401,403,429, any5xxanswer, andNone.
may_use_copy(age_hours, reachable)says whether a cached copy may still answer questions: when it is less than 24 hours old, or, whatever its age, while the file cannot be reached (reachableisFalse).
The sample tests try the edges of every range, such as 299, 300 and 500.
Starter code · policy.py
def policy(status, redirects=0):
""""follow the rules", "no rules" or "disallow everything" for the answer to the robots.txt request."""
# Replace this: a failed request is not a licence to crawl.
return "follow the rules"
def may_use_copy(age_hours, reachable):
"""True while a cached robots.txt may still answer: under 24 hours old, or while the file is unreachable."""
# Replace this: a copy does not last for ever.
return TrueThe sample tests · test_policy.py
from policy import may_use_copy, policy
def test_success_follows_the_rules():
"""2xx: follow the rules, at the edges of the range too"""
for status in (200, 204, 299):
assert policy(status) == "follow the rules", f"{status} is a success"
assert policy(200, redirects=5) == "follow the rules", "RFC 9309: follow at least five redirects"
def test_too_many_redirects_mean_unavailable():
"""more than 5 redirects: the file is treated as unavailable"""
assert policy(200, redirects=6) == "no rules"
def test_missing_file_means_no_rules():
"""404 and other 4xx answers: unavailable, so no rules apply"""
for status in (400, 404, 410, 499):
assert policy(status) == "no rules", f"{status}: the file is unavailable"
def test_refusals_and_rate_limits_disallow_everything():
"""401, 403 and 429: stop for now"""
for status in (401, 403, 429):
assert policy(status) == "disallow everything", f"{status}: a careful collector stops"
def test_server_and_network_errors_disallow_everything():
"""5xx and network errors: unreachable, so complete disallow"""
for status in (500, 503, 599, None):
assert policy(status) == "disallow everything", f"{status}: the file is unreachable"
assert policy(300) != "follow the rules", "a 3xx that was not followed is not a file to obey"
def test_a_copy_lasts_less_than_a_day():
"""fresh for under 24 hours; older only while the file is unreachable"""
assert may_use_copy(0, True) and may_use_copy(23.9, True)
assert not may_use_copy(24, True), "at 24 hours the copy must be fetched again"
assert not may_use_copy(30, True)
assert may_use_copy(30, False), "while the file is unreachable the old copy may stay"A hint
Check None and the redirect count first, then the ranges with comparisons such as 200 <= status < 300. Put the three exceptions, 401, 403 and 429, before the general 4xx case. For may_use_copy, a copy is usable when age_hours < 24 or when the file is not reachable.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
The self-check comes last: what the file is for, groups, longest rules, status codes, the day-long cache, Crawl-delay and parsers, seven ideas in all, and it opens on whichever of them your earlier answers left shakiest:
Practice that adapts to you
Practise this lesson
15 questions about 7 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
Warm-up (fresher): What does robots.txt control, and what does it not control?
robots.txt is a request from a site to automated clients: it says, group by product token, which paths they should
not fetch, and RFC 9309 defines how to read it. It controls nothing by force. Its rules are not access
authorization, a browser ignores them, and a client that disregards them still gets the pages, so anything private
needs a login or another real control on the server. It does not say how the data may be used once fetched, which is
for the terms and the licence, and it does not set a rate in the standard: Crawl-delay is an extension. Because the
file is public, listing a path in it also tells everyone the path exists.
Two harder questions start from this lesson: a robots.txt that answers 503 for hours, and a bot that began fetching
pages its own group forbids. The module’s question page has the first step and the follow-ups of each:
Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.
-
Mid to senior
For six hours, every request for a site's robots.txt has answered 503. Your crawler holds a copy of the file from yesterday and 40,000 of the site's addresses in its queue. What does it do now, and what changes if the 503s go on for weeks?
-
Fresher to mid
After a refactor, your crawler starts fetching pages under /prices/, which its own group in a site's robots.txt disallows, while every other rule seems to be followed. How do you find the cause, and which test stops it coming back?
Key takeaways
- robots.txt asks crawlers to stay out of some paths; it is no access control, and it covers only the scheme, host and port it is served from.
- The product token picks the group: every group that names it is merged,
*applies only when none does, and a named group replaces the*rules entirely. - In the group, the longest matching rule decides and
Allowwins a tie;*and a final$are wildcards, paths are compared percent-encoded, and/robots.txtis always allowed. - A
5xxor a network error means disallow everything; a404means no rules; and a careful collector also stops for401,403and429, which the standard leaves to its judgement. - Keep one copy per site for at most 24 hours, longer only while the file cannot be reached, and read up to 500 KiB of it.
Crawl-delayis outside the standard, andurllib.robotparserdrops a delay with decimals; check every parser’s answers, including the old one in the browser runner, before trusting it.
Terms in this lesson
- Crawl-delay
- A robots.txt line that asks a bot to wait a number of seconds between requests. It is not part of RFC 9309, and parsers differ in how they read it.
- product token
- The short name a bot is known by in robots.txt, made of letters, underscores and hyphens, such as NimbuBot. It is also the first word of the bot's User-Agent header.
- robots.txt
- A plain text file at /robots.txt in which a site asks automated clients to stay out of some paths, group by product token. RFC 9309 defines it; it is a request, not access control.
References
- RFC 9309: Robots Exclusion Protocol (IETF)
- urllib.robotparser: Parser for robots.txt (Python 3.14) (Python Software Foundation)
- CPython source: Lib/urllib/robotparser.py at v3.14.8 (Python Software Foundation)
- CPython source: Lib/urllib/robotparser.py at v3.14.2 (Python Software Foundation)
- CPython source: Lib/urllib/robotparser.py at v3.14.5 (Python Software Foundation)
- CPython source: Lib/urllib/robotparser.py at v3.14.7 (Python Software Foundation)
- How Google interprets the robots.txt specification (Google)
- Protego, a pure-Python robots.txt parser (Scrapy project)
- robots-parser for Node.js (Sam Clarke (robots-parser))
- Wikimedia Foundation User-Agent Policy (Wikimedia Foundation)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress