Web Scraping & Browser Automation (Python and JavaScript) Module 6 – Scrapy 2.19 spiders, pipelines and operations
Scrapy politeness: robots, throttle, cache
Make a Scrapy 2.19 crawl obey robots.txt, pace itself with delays, jitter and AutoThrottle, honour Crawl-delay and Retry-After, and revalidate its HTTP cache.
What you will learn
- Configure robots.txt obedience and read the stats that show what the robots.txt middleware refused
- Set a crawl's pace with DOWNLOAD_DELAY, its jitter and AutoThrottle, and measure the gaps it produces
- Fix the two gaps in Scrapy 2.19.0's robots.txt handling with a tested middleware
- Apply Crawl-delay and Retry-After to the download slot, since Scrapy 2.19 does neither on its own
- Choose an HTTP cache policy that revalidates instead of serving stale pages
Before you start
What you’ll learn · Before you start
What you will learn
- Configure robots.txt obedience and read the stats that show what the robots.txt middleware refused
- Set a crawl's pace with DOWNLOAD_DELAY, its jitter and AutoThrottle, and measure the gaps it produces
- Fix the two gaps in Scrapy 2.19.0's robots.txt handling with a tested middleware
- Apply Crawl-delay and Retry-After to the download slot, since Scrapy 2.19 does neither on its own
- Choose an HTTP cache policy that revalidates instead of serving stale pages
Before you start
On this page
The practice shop’s robots.txt asks NimbuBot to stay out of /account/, /people/ and /prices/ and to wait 2
seconds between requests. A Scrapy 2.19 crawl with the template’s settings and five start URLs fetched 2 of the 3
pages the file forbids, and it kept its own delay of 1 second, not 2.
This lesson makes a crawl keep what robots.txt, Crawl-delay, Retry-After and HTTP caching ask of it, and measures each promise on the shop’s own clock.
Scrapy obeys robots.txt, except for requests sent at the start
With ROBOTSTXT_OBEY = True, Scrapy’s robots.txt middleware asks for a site’s robots.txt: A plain text file at /robots.txt in which a site asks automated clients to stay out of some paths, group by product token. RFC 9309 defines it; it is a request, not access control. More in “robots.txt for your bot: RFC 9309 rules” before its first request
there, reads it with Protego and drops every request the rules disallow, counting them in robotstxt/forbidden. The
group it follows is chosen by the User-Agent, or by ROBOTSTXT_USER_AGENT when that is set. The next program sends
the same five requests three times: as NimbuBot all at once, as NimbuBot one at a time, and with Scrapy’s own
User-Agent:
polite/robots_obey.py
"""robots.txt in Scrapy 2.19: the group a User-Agent gets, the requests it refuses, and requests sent all at once."""
import scrapy
from twisted.internet.task import react
from scrapy import signals
from scrapy.crawler import AsyncCrawlerRunner
from scrapy.utils.defer import deferred_f_from_coro_f
from scrapy.utils.reactor import install_reactor
import nimbu_lab
PATHS = ["/products/grid/?page=1", "/prices/today", "/account/orders", "/people/asha", "/products/grid/?page=2"]
BOT = "NimbuBot/1.0 (+https://bot.example.org/; bots@example.org)"
SETTINGS = {"ROBOTSTXT_OBEY": True, "CONCURRENT_REQUESTS_PER_DOMAIN": 1, "DOWNLOAD_DELAY": 1,
"TELNETCONSOLE_ENABLED": False, "REMOTE_CONTROL_ENABLED": False, "LOG_ENABLED": False,
"DOWNLOAD_HANDLERS": {"https": "nimbu_lab.LabHandler"}}
agent, crawl_delays = [""], []
class Visit(scrapy.Spider):
name = "visit"
async def start(self):
for path in PATHS: # five start requests, yielded at once
yield scrapy.Request(f"https://{nimbu_lab.HOST}{path}")
def parse(self, response):
pass
def robots_parsed(robotparser, request, **kwargs):
crawl_delays.append(robotparser.crawl_delay(agent[0])) # Scrapy 2.18 and later: every parser has crawl_delay()
async def crawl(_):
runs = [("NimbuBot, five start requests at once", {"USER_AGENT": BOT}),
("NimbuBot, CONCURRENT_REQUESTS = 1", {"USER_AGENT": BOT, "CONCURRENT_REQUESTS": 1}),
("Scrapy's own User-Agent, CONCURRENT_REQUESTS = 1", {"CONCURRENT_REQUESTS": 1})]
for label, settings in runs:
first = len(nimbu_lab.LOG)
runner = AsyncCrawlerRunner({**SETTINGS, **settings})
crawler = runner.create_crawler(Visit)
agent[0] = crawler.settings["USER_AGENT"]
crawler.signals.connect(robots_parsed, signal=signals.robots_parsed)
await runner.crawl(crawler)
print(f"{label}:")
print(" the shop received ", ", ".join(e["path"] for e in nimbu_lab.LOG[first:]))
print(" robotstxt/forbidden ", crawler.stats.get_value("robotstxt/forbidden", 0))
print(" the group's Crawl-delay", crawl_delays[-1], "s; Scrapy waited DOWNLOAD_DELAY, 1 s, all the same")
if __name__ == "__main__":
install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")
react(deferred_f_from_coro_f(crawl))Output
NimbuBot, five start requests at once: the shop received /robots.txt, /products/grid/?page=1, /account/orders, /people/asha, /products/grid/?page=2 robotstxt/forbidden 1 the group's Crawl-delay 2.0 s; Scrapy waited DOWNLOAD_DELAY, 1 s, all the same NimbuBot, CONCURRENT_REQUESTS = 1: the shop received /robots.txt, /products/grid/?page=1, /products/grid/?page=2 robotstxt/forbidden 3 the group's Crawl-delay 2.0 s; Scrapy waited DOWNLOAD_DELAY, 1 s, all the same Scrapy's own User-Agent, CONCURRENT_REQUESTS = 1: the shop received /robots.txt, /products/grid/?page=1, /prices/today, /products/grid/?page=2 robotstxt/forbidden 2 the group's Crawl-delay 3.0 s; Scrapy waited DOWNLOAD_DELAY, 1 s, all the same
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 robots_obey.py
nimbu_lab.py (the shop) · polite/nimbu_lab.py
"""Nimbu Bazaar, the practice shop of this track, for Scrapy. Every product and price in it is invented.
app() is the shop, a WSGI application (PEP 3333). LabHandler is a Scrapy download handler: with the setting
DOWNLOAD_HANDLERS = {"https": "nimbu_lab.LabHandler"} every request of a crawl goes through Scrapy's whole engine
and then to app() inside the same process, so no request leaves the program. LOG is the shop's access log, with the
moment each request arrived, so the examples can measure the gaps between requests.
The settings below the products are this lesson's knobs: the shop's robots.txt and its status, a limit that answers
429 with Retry-After, the day (on day 2 some prices change) and pages that answer slowly.
"""
import asyncio
import hashlib
import io
import time
from urllib.parse import parse_qs, urlsplit
HOST = "bazaar.nimbu.test"
ROBOTS = """\
# Nimbu Bazaar, the practice shop: every product here is invented.
User-agent: *
Disallow: /account/
Disallow: /people/
Crawl-delay: 3
User-agent: NimbuBot
Disallow: /account/
Disallow: /people/
Disallow: /prices/
Crawl-delay: 2
"""
ROBOTS_STATUS = 200 # 503: robots.txt cannot be read right now
LIMIT_EVERY = 0 # n > 0: every n-th listing request is answered 429 with Retry-After
RETRY_AFTER = 3 # seconds the shop asks a refused client to wait
DAY = 1 # on day 2 the products of pages 3 and 7 cost 5 % more
LATENCY = 0.35 # seconds the shop takes to answer a page
SLOW_PAGES = set() # listing pages that take 2 s instead
CACHE_CONTROL = "max-age=0" # what listing pages say: stale at once, so ask with the ETag before using a copy
KINDS = ["Basmati rice", "Toor dal", "Green tea", "Jaggery", "Turmeric", "Mustard oil", "Poha", "Rock salt",
"Cashews", "Cardamom", "Ragi flour", "Ghee", "Honey", "Peanuts", "Coffee", "Raisins", "Sesame seeds",
"Tamarind", "Cumin", "Black pepper", "Rolled oats", "Cloves", "Fennel", "Moong dal", "Chana dal",
"Wheat flour", "Semolina", "Coriander", "Red chilli", "Almonds", "Dates", "Coconut oil", "Brown rice",
"Millet", "Flax seeds", "Cinnamon", "Ginger powder", "Vermicelli", "Masoor dal", "Sago"]
SIZES = [250, 500, 1000, 2000, 5000, 10000] # grams
LOG = [] # the access log: one entry per request, in the order the shop received them
STARTED = time.monotonic()
refused_until = [0.0] # the end of the latest Retry-After window the shop gave
listing_requests = [0]
def products():
"""The 240 products: 40 kinds in 6 pack sizes, each price in paise (100 paise = 1 rupee)."""
out = []
for k, kind in enumerate(KINDS):
per_kg = 60 + (k * 37) % 900 # rupees a kilogram
for grams in SIZES:
size = f"{grams // 1000} kg" if grams >= 1000 else f"{grams} g"
paise = per_kg * grams // 10 + 1000
if DAY == 2 and len(out) // 24 + 1 in (3, 7) and len(out) % 4 == 0:
paise = paise * 105 // 100
out.append({"sku": f"NB-{len(out) + 1:04d}", "name": f"{kind} {size}", "paise": paise})
return out
def rupees(paise):
return f"₹{paise // 100:,}.{paise % 100:02d}"
def grid(page):
cards = "".join(
f'<li class="product" data-sku="{p["sku"]}">\n <a class="name" href="/products/{p["sku"]}/">{p["name"]}</a>\n'
f' <span class="price">{rupees(p["paise"])}</span>\n</li>\n'
for p in products()[(page - 1) * 24:page * 24])
pages = " ".join(f'<a href="?page={n}">{n}</a>' for n in range(1, 11))
return (f'<!doctype html>\n<html lang="en"><head><meta charset="utf-8">'
f"<title>Products, page {page} · Nimbu Bazaar</title></head><body>\n"
f'<main id="catalogue"><h1>Products</h1>\n<ul class="grid">\n{cards}</ul>\n'
f'<nav class="pages">{pages}</nav></main></body></html>\n')
def app(environ, start_response):
"""The shop: answers one request and writes it into LOG."""
now = round(time.monotonic() - STARTED, 3)
path, query = environ["PATH_INFO"], parse_qs(environ.get("QUERY_STRING", ""))
page = query.get("page", ["1"])[0]
extra, body, kind = [], "", "text/plain"
if environ.get("HTTP_HOST") != HOST:
status, body = "421 Misdirected Request", "not a Nimbu Bazaar address\n"
elif path == "/robots.txt" and ROBOTS_STATUS != 200:
status, kind, body = "503 Service Unavailable", "text/html", "<html><body><h1>Back soon</h1></body></html>\n"
elif path == "/robots.txt":
status, body, extra = "200 OK", ROBOTS, [("Cache-Control", "max-age=86400")]
elif path == "/products/grid/" and page in [str(n) for n in range(1, 11)]:
listing_requests[0] += 1
if LIMIT_EVERY and listing_requests[0] % LIMIT_EVERY == 0:
refused_until[0] = now + RETRY_AFTER
status, body, extra = "429 Too Many Requests", "slow down, please\n", [("Retry-After", str(RETRY_AFTER))]
else:
html = grid(int(page))
tag = '"' + hashlib.sha256(html.encode()).hexdigest()[:16] + '"'
extra = [("ETag", tag), ("Cache-Control", CACHE_CONTROL)]
if environ.get("HTTP_IF_NONE_MATCH") == tag:
status = "304 Not Modified"
else:
status, kind, body = "200 OK", "text/html", html
else:
status, body = "404 Not Found", "no such page\n"
data = body.encode()
headers = [("Content-Type", f"{kind}; charset=utf-8"), ("Content-Length", str(len(data))), *extra]
target = path + ("?" + environ["QUERY_STRING"] if environ.get("QUERY_STRING") else "")
LOG.append({"at": now, "path": target, "status": int(status[:3]), "conditional": "HTTP_IF_NONE_MATCH" in environ,
"bytes": len(data), "user_agent": environ.get("HTTP_USER_AGENT", "")})
start_response(status, headers)
return [data]
class LabHandler:
"""A Scrapy download handler (DOWNLOAD_HANDLERS) that answers every request from app(), with no network."""
lazy = False
def __init__(self, crawler):
self.crawler = crawler
@classmethod
def from_crawler(cls, crawler):
return cls(crawler)
async def download_request(self, request):
from scrapy.http import Headers
from scrapy.responsetypes import responsetypes
parts = urlsplit(request.url)
environ = {"REQUEST_METHOD": request.method, "PATH_INFO": parts.path or "/", "QUERY_STRING": parts.query,
"SERVER_NAME": HOST, "SERVER_PORT": "443", "SERVER_PROTOCOL": "HTTP/1.1",
"wsgi.url_scheme": "https", "wsgi.input": io.BytesIO(request.body), "HTTP_HOST": parts.netloc}
for name, values in request.headers.items():
environ["HTTP_" + name.decode().upper().replace("-", "_")] = b", ".join(values).decode()
got, started = {}, time.monotonic()
body = b"".join(app(environ, lambda status, hdrs, exc_info=None: got.update(status=status, headers=hdrs)))
slow = parts.path == "/products/grid/" and parse_qs(parts.query).get("page", [""])[0] in {str(p) for p in SLOW_PAGES}
await asyncio.sleep(2.0 if slow else LATENCY)
request.meta["download_latency"] = time.monotonic() - started # as Scrapy's HTTP handlers record it
headers = Headers(got["headers"])
respcls = responsetypes.from_args(headers=headers, url=request.url, body=body)
return respcls(url=request.url, status=int(got["status"][:3]), headers=headers, body=body, request=request)
async def close(self):
pass
def gaps(log):
"""Seconds between the starts of consecutive requests in a part of LOG."""
return [round(b["at"] - a["at"], 2) for a, b in zip(log, log[1:])]One at a time, NimbuBot’s group refused /prices/today, /account/orders and /people/asha, while Scrapy’s own name
fell into the * group, which allows /prices/. All at once, the first crawl refused only one of the three. Step
through its first second:
The drawing shows the whole scene; its 7 steps are listed below.
Every step as text
- Five requests at once. start() yields page 1 of the listing, /prices/today, /account/orders, /people/asha and page 2. NimbuBot's group disallows the middle three. Up to sixteen requests may be on their way at once, so all five reach the middleware together.
- robots.txt first. Page 1 arrives first, and the middleware has no rules for this host yet. It asks the shop for /robots.txt and holds page 1 until the file is read.
- Four more wait. The other four requests arrive while robots.txt is still on its way. Each of them waits for the same download.
- The rules arrive. The shop answers 200 with its rules, and the middleware builds one parser for the host. Page 1 is allowed and goes on.
- The first waiter is refused. The parser reaches the first waiting request, /prices/today, which NimbuBot's group disallows, so it is dropped and counted in robotstxt/forbidden.
- The others pass unchecked. The three other waiting requests receive no parser at all, which the middleware treats as a site without rules. /account/orders and /people/asha reach the shop although robots.txt disallows them, and page 2 goes with them.
- Later requests are checked. The parser is now stored for the host, so every later request, a followed link or a retry, is checked against it. Only the requests that waited during the download slipped through.
Requests that wait for robots.txt are not checked in Scrapy 2.19.0
Requests that reach the robots.txt middleware while a site’s robots.txt is still downloading are let through
unchecked, all but the first of them. For a crawl of one site, CONCURRENT_REQUESTS = 1 keeps them out, as the
second crawl shows. Test it with a disallowed start URL and the robotstxt/forbidden count.
Checkpoint
Check: robots.txt obedience and the start-up gap
A download slot sets the pace: delay, jitter and AutoThrottle
Scrapy paces each download slot: The queue in Scrapy's downloader for the requests of one host, by default. DOWNLOAD_DELAY, its jitter and CONCURRENT_REQUESTS_PER_DOMAIN apply to each slot, so every host is paced on its own., one per host by default: DOWNLOAD_DELAY seconds
between two requests, varied by DOWNLOAD_DELAY_JITTER, which Scrapy 2.19 sets to ±50 %. The
AutoThrottle: A Scrapy extension, off by default, that changes each download slot's delay as a crawl runs: it starts at 5 s and moves halfway towards the site's latency after each response, never below DOWNLOAD_DELAY. extension is off by default; when it is on, it starts each slot at 5 s and
moves the delay towards the site’s latency, never below DOWNLOAD_DELAY. Measure both:
polite/pace.py
"""The pace of a Scrapy crawl, measured on the shop's clock: DOWNLOAD_DELAY with its jitter, then AutoThrottle."""
import random
import scrapy
from twisted.internet.task import react
from scrapy import signals
from scrapy.crawler import AsyncCrawlerRunner
from scrapy.utils.defer import deferred_f_from_coro_f
from scrapy.utils.reactor import install_reactor
import nimbu_lab
SETTINGS = {"ROBOTSTXT_OBEY": True, "CONCURRENT_REQUESTS": 1, "CONCURRENT_REQUESTS_PER_DOMAIN": 1,
"DOWNLOAD_DELAY": 1, "USER_AGENT": "NimbuBot/1.0 (+https://bot.example.org/; bots@example.org)",
"TELNETCONSOLE_ENABLED": False, "REMOTE_CONTROL_ENABLED": False, "LOG_ENABLED": False,
"DOWNLOAD_HANDLERS": {"https": "nimbu_lab.LabHandler"}}
seen, crawlers = [], []
class EightPages(scrapy.Spider):
name = "eight_pages"
async def start(self):
for n in range(1, 9):
yield scrapy.Request(f"https://{nimbu_lab.HOST}/products/grid/?page={n}")
def parse(self, response):
pass
def response_received(response, request, **kwargs):
slot = crawlers[-1].engine.downloader.slots[request.meta["download_slot"]]
seen.append((response.url.split("/")[-1] or "robots.txt", request.meta["download_latency"], slot.delay))
async def crawl(_):
random.seed(2019) # Scrapy draws its jitter from random; a seed makes the recorded run repeat
print("1. DOWNLOAD_DELAY = 1 with DOWNLOAD_DELAY_JITTER at its default, 0.5")
first = len(nimbu_lab.LOG)
await AsyncCrawlerRunner(SETTINGS).crawl(EightPages)
gaps = nimbu_lab.gaps(nimbu_lab.LOG[first:])
print(f" {len(gaps)} gaps between the 9 requests: shortest {min(gaps):.1f} s, longest {max(gaps):.1f} s, "
f"average {sum(gaps) / len(gaps):.1f} s")
print("2. AutoThrottle with its defaults, DOWNLOAD_DELAY = 1 as its floor; pages 5 to 8 take 2 s to answer")
nimbu_lab.SLOW_PAGES = {5, 6, 7, 8}
runner = AsyncCrawlerRunner({**SETTINGS, "AUTOTHROTTLE_ENABLED": True})
crawler = runner.create_crawler(EightPages)
crawlers.append(crawler)
crawler.signals.connect(response_received, signal=signals.response_received)
await runner.crawl(crawler)
print(" response latency delay AutoThrottle set for the next request")
for name, latency, delay in seen:
print(f" {name:<17} {latency:5.1f} s {delay:4.1f} s")
if __name__ == "__main__":
install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")
react(deferred_f_from_coro_f(crawl))Output
1. DOWNLOAD_DELAY = 1 with DOWNLOAD_DELAY_JITTER at its default, 0.5 8 gaps between the 9 requests: shortest 0.9 s, longest 1.5 s, average 1.3 s 2. AutoThrottle with its defaults, DOWNLOAD_DELAY = 1 as its floor; pages 5 to 8 take 2 s to answer response latency delay AutoThrottle set for the next request robots.txt 0.4 s 2.7 s ?page=1 0.4 s 1.5 s ?page=2 0.4 s 1.0 s ?page=3 0.4 s 1.0 s ?page=4 0.4 s 1.0 s ?page=5 2.0 s 2.0 s ?page=6 2.0 s 2.0 s ?page=7 2.0 s 2.0 s ?page=8 2.0 s 2.0 s
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 pace.py
nimbu_lab.py (the shop) · polite/nimbu_lab.py
"""Nimbu Bazaar, the practice shop of this track, for Scrapy. Every product and price in it is invented.
app() is the shop, a WSGI application (PEP 3333). LabHandler is a Scrapy download handler: with the setting
DOWNLOAD_HANDLERS = {"https": "nimbu_lab.LabHandler"} every request of a crawl goes through Scrapy's whole engine
and then to app() inside the same process, so no request leaves the program. LOG is the shop's access log, with the
moment each request arrived, so the examples can measure the gaps between requests.
The settings below the products are this lesson's knobs: the shop's robots.txt and its status, a limit that answers
429 with Retry-After, the day (on day 2 some prices change) and pages that answer slowly.
"""
import asyncio
import hashlib
import io
import time
from urllib.parse import parse_qs, urlsplit
HOST = "bazaar.nimbu.test"
ROBOTS = """\
# Nimbu Bazaar, the practice shop: every product here is invented.
User-agent: *
Disallow: /account/
Disallow: /people/
Crawl-delay: 3
User-agent: NimbuBot
Disallow: /account/
Disallow: /people/
Disallow: /prices/
Crawl-delay: 2
"""
ROBOTS_STATUS = 200 # 503: robots.txt cannot be read right now
LIMIT_EVERY = 0 # n > 0: every n-th listing request is answered 429 with Retry-After
RETRY_AFTER = 3 # seconds the shop asks a refused client to wait
DAY = 1 # on day 2 the products of pages 3 and 7 cost 5 % more
LATENCY = 0.35 # seconds the shop takes to answer a page
SLOW_PAGES = set() # listing pages that take 2 s instead
CACHE_CONTROL = "max-age=0" # what listing pages say: stale at once, so ask with the ETag before using a copy
KINDS = ["Basmati rice", "Toor dal", "Green tea", "Jaggery", "Turmeric", "Mustard oil", "Poha", "Rock salt",
"Cashews", "Cardamom", "Ragi flour", "Ghee", "Honey", "Peanuts", "Coffee", "Raisins", "Sesame seeds",
"Tamarind", "Cumin", "Black pepper", "Rolled oats", "Cloves", "Fennel", "Moong dal", "Chana dal",
"Wheat flour", "Semolina", "Coriander", "Red chilli", "Almonds", "Dates", "Coconut oil", "Brown rice",
"Millet", "Flax seeds", "Cinnamon", "Ginger powder", "Vermicelli", "Masoor dal", "Sago"]
SIZES = [250, 500, 1000, 2000, 5000, 10000] # grams
LOG = [] # the access log: one entry per request, in the order the shop received them
STARTED = time.monotonic()
refused_until = [0.0] # the end of the latest Retry-After window the shop gave
listing_requests = [0]
def products():
"""The 240 products: 40 kinds in 6 pack sizes, each price in paise (100 paise = 1 rupee)."""
out = []
for k, kind in enumerate(KINDS):
per_kg = 60 + (k * 37) % 900 # rupees a kilogram
for grams in SIZES:
size = f"{grams // 1000} kg" if grams >= 1000 else f"{grams} g"
paise = per_kg * grams // 10 + 1000
if DAY == 2 and len(out) // 24 + 1 in (3, 7) and len(out) % 4 == 0:
paise = paise * 105 // 100
out.append({"sku": f"NB-{len(out) + 1:04d}", "name": f"{kind} {size}", "paise": paise})
return out
def rupees(paise):
return f"₹{paise // 100:,}.{paise % 100:02d}"
def grid(page):
cards = "".join(
f'<li class="product" data-sku="{p["sku"]}">\n <a class="name" href="/products/{p["sku"]}/">{p["name"]}</a>\n'
f' <span class="price">{rupees(p["paise"])}</span>\n</li>\n'
for p in products()[(page - 1) * 24:page * 24])
pages = " ".join(f'<a href="?page={n}">{n}</a>' for n in range(1, 11))
return (f'<!doctype html>\n<html lang="en"><head><meta charset="utf-8">'
f"<title>Products, page {page} · Nimbu Bazaar</title></head><body>\n"
f'<main id="catalogue"><h1>Products</h1>\n<ul class="grid">\n{cards}</ul>\n'
f'<nav class="pages">{pages}</nav></main></body></html>\n')
def app(environ, start_response):
"""The shop: answers one request and writes it into LOG."""
now = round(time.monotonic() - STARTED, 3)
path, query = environ["PATH_INFO"], parse_qs(environ.get("QUERY_STRING", ""))
page = query.get("page", ["1"])[0]
extra, body, kind = [], "", "text/plain"
if environ.get("HTTP_HOST") != HOST:
status, body = "421 Misdirected Request", "not a Nimbu Bazaar address\n"
elif path == "/robots.txt" and ROBOTS_STATUS != 200:
status, kind, body = "503 Service Unavailable", "text/html", "<html><body><h1>Back soon</h1></body></html>\n"
elif path == "/robots.txt":
status, body, extra = "200 OK", ROBOTS, [("Cache-Control", "max-age=86400")]
elif path == "/products/grid/" and page in [str(n) for n in range(1, 11)]:
listing_requests[0] += 1
if LIMIT_EVERY and listing_requests[0] % LIMIT_EVERY == 0:
refused_until[0] = now + RETRY_AFTER
status, body, extra = "429 Too Many Requests", "slow down, please\n", [("Retry-After", str(RETRY_AFTER))]
else:
html = grid(int(page))
tag = '"' + hashlib.sha256(html.encode()).hexdigest()[:16] + '"'
extra = [("ETag", tag), ("Cache-Control", CACHE_CONTROL)]
if environ.get("HTTP_IF_NONE_MATCH") == tag:
status = "304 Not Modified"
else:
status, kind, body = "200 OK", "text/html", html
else:
status, body = "404 Not Found", "no such page\n"
data = body.encode()
headers = [("Content-Type", f"{kind}; charset=utf-8"), ("Content-Length", str(len(data))), *extra]
target = path + ("?" + environ["QUERY_STRING"] if environ.get("QUERY_STRING") else "")
LOG.append({"at": now, "path": target, "status": int(status[:3]), "conditional": "HTTP_IF_NONE_MATCH" in environ,
"bytes": len(data), "user_agent": environ.get("HTTP_USER_AGENT", "")})
start_response(status, headers)
return [data]
class LabHandler:
"""A Scrapy download handler (DOWNLOAD_HANDLERS) that answers every request from app(), with no network."""
lazy = False
def __init__(self, crawler):
self.crawler = crawler
@classmethod
def from_crawler(cls, crawler):
return cls(crawler)
async def download_request(self, request):
from scrapy.http import Headers
from scrapy.responsetypes import responsetypes
parts = urlsplit(request.url)
environ = {"REQUEST_METHOD": request.method, "PATH_INFO": parts.path or "/", "QUERY_STRING": parts.query,
"SERVER_NAME": HOST, "SERVER_PORT": "443", "SERVER_PROTOCOL": "HTTP/1.1",
"wsgi.url_scheme": "https", "wsgi.input": io.BytesIO(request.body), "HTTP_HOST": parts.netloc}
for name, values in request.headers.items():
environ["HTTP_" + name.decode().upper().replace("-", "_")] = b", ".join(values).decode()
got, started = {}, time.monotonic()
body = b"".join(app(environ, lambda status, hdrs, exc_info=None: got.update(status=status, headers=hdrs)))
slow = parts.path == "/products/grid/" and parse_qs(parts.query).get("page", [""])[0] in {str(p) for p in SLOW_PAGES}
await asyncio.sleep(2.0 if slow else LATENCY)
request.meta["download_latency"] = time.monotonic() - started # as Scrapy's HTTP handlers record it
headers = Headers(got["headers"])
respcls = responsetypes.from_args(headers=headers, url=request.url, body=body)
return respcls(url=request.url, status=int(got["status"][:3]), headers=headers, body=body, request=request)
async def close(self):
pass
def gaps(log):
"""Seconds between the starts of consecutive requests in a part of LOG."""
return [round(b["at"] - a["at"], 2) for a, b in zip(log, log[1:])]The delay of 1 s gave gaps from 0.9 to 1.5 s, 1.3 s on average. AutoThrottle went from 2.7 to 1.5 s, stopped at the 1.0 s floor while the shop answered fast, and rose to 2.0 s when it slowed. Follow the slot:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- One slot per host. Every request to bazaar.nimbu.test goes into the same download slot. DOWNLOAD_DELAY and CONCURRENT_REQUESTS_PER_DOMAIN apply to each slot, so two hosts are paced apart.
- A jittered delay. Before each request the slot waits DOWNLOAD_DELAY, varied by up to 50 % either way, so with a delay of 1 s a single wait lies between 0.5 and 1.5 s.
- Measured, not assumed. Scrapy draws a new random delay each time it looks at the queue, and a longer draw postpones the request again. The 8 gaps of the run ran from 0.9 to 1.5 s, 1.3 s on average.
- AutoThrottle starts slow. With AUTOTHROTTLE_ENABLED, the slot starts at AUTOTHROTTLE_START_DELAY, 5 s. After each response it sets a new delay halfway between the old one and the latency, never below DOWNLOAD_DELAY.
- Down to the floor. The shop answered in about 0.4 s, so the delay fell from 2.7 to 1.5 s and then stopped at 1.0 s, the DOWNLOAD_DELAY floor.
- A slower host, a longer delay. From page 5 on the shop took 2 s to answer. A delay never stays below the latency, so it went straight up to 2.0 s and the crawl slowed down with the shop.
Four promises take more than settings, and each has a part of its own below. Scrapy 2.19 reads Crawl-delay: A robots.txt line that asks a bot to wait a number of seconds between requests. It is not part of RFC 9309, and parsers differ in how they read it. More in “robots.txt for your bot: RFC 9309 rules” but never applies it. When robots.txt answers 503, Scrapy treats the site as having no rules, where RFC 9309 says to assume everything is disallowed. RetryMiddleware retries a 429 without waiting for its Retry-After. And the HTTP cache’s default policy, DummyPolicy, replays stored pages without asking whether they changed.
Key idea
Pace per host: a delay that robots.txt and the site’s answers can only lengthen, checked on the site’s clock, not on your settings.
Checkpoint
Check: Download slots, delays, jitter and AutoThrottle
Both robots.txt gaps closed by one middleware, reproduced Pro: opens with a Pro, Premium or Ultimate pass
Scrapy 2.19.0 lets some requests past robots.txt in two situations: while the file is downloading, and when the site cannot serve it. This part reproduces both on the practice shop and replaces the middleware with one that closes them.
In this part:
- Why the waiting requests get no rules, and the fix
- A robots.txt that answers 503 means “disallow everything”
- Wiring the fix in, and knowing when to remove it
1 runnable example (middlewares.py, nimbu_lab.py (the shop), robots_race.py) · about 4 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Crawl-delay and Retry-After applied to the download slot, measured Pro: opens with a Pro, Premium or Ultimate pass
Crawl-delay and Retry-After both ask a client to wait, and Scrapy 2.19 applies neither. This part adds an extension and a downloader middleware that change the host’s download slot, and measures the gaps against both requests.
In this part:
- Crawl-delay as the shortest gap, not the average
- Retry-After: hold the whole host, then retry
- Practise: audit a crawl’s access log
- Audit a crawl's access log for Crawl-delay and Retry-After
- Results of the sample tests
2 runnable examples (crawl_delay.py, extensions.py, nimbu_lab.py (the shop), retry_after.py) · 1 exercise with sample tests · about 5 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Scrapy's HTTP cache: replay, revalidate, and a no-cache surprise Pro: opens with a Pro, Premium or Ultimate pass
Scrapy’s HTTP cache can make a re-run cost almost nothing or make it report last week’s data; the policy decides which. This part crawls the shop on two days under each policy and counts requests, bytes and stale prices.
In this part:
- DummyPolicy against RFC2616Policy over two days, measured
- Pages marked no-cache are downloaded in full
- Where the cache lives, and what it keeps
2 runnable examples (Python, http_cache.py, nimbu_lab.py (the shop)) · about 4 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Practise politeness in Scrapy
The practice questions cover robots.txt and the pace of a crawl, starting with the idea your answers have left weakest:
Practice that adapts to you
Practise this lesson
6 questions about 5 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
Warm-up (fresher): What is the difference between DOWNLOAD_DELAY and AutoThrottle?
DOWNLOAD_DELAY is a fixed wait between two requests to one host, varied by its jitter, ±50 % by default.
AutoThrottle, off unless enabled, changes that wait as the crawl runs: it starts at 5 s and moves the delay towards the
host’s latency, so a host that slows down is asked less often. It never goes below DOWNLOAD_DELAY, so set that to the
shortest delay the site’s terms allow.
Three harder questions grow out of this lesson: disallowed requests in the first second of a crawl, a Crawl-delay that the site’s owner says you ignore, and a cache that reported last week’s prices:
-
Mid to senior
Your Scrapy 2.19 crawl has ROBOTSTXT_OBEY = True, yet a site's owner shows you requests to disallowed paths in the first second of every run, and none later. What is happening, and how do you fix it and prove the fix?
-
Mid to senior
A site's robots.txt says Crawl-delay: 5. Your Scrapy spider sets DOWNLOAD_DELAY = 5, yet the site's owner reports gaps of 2.5 to 3 seconds in their log. Explain the gaps, and change the spider so that no gap is shorter than 5 seconds.
-
Fresher to mid
You turned on Scrapy's HTTP cache to make reruns fast during development, and later the nightly price crawl reported last week's prices for a whole week. Why, and how do you configure the cache so that it saves bandwidth without serving stale data?
Key takeaways
ROBOTSTXT_OBEYreads robots.txt before a site’s first request and drops what the rules forbid, but in Scrapy 2.19.0 requests sent while robots.txt downloads skip the check;CONCURRENT_REQUESTS = 1closes the gap for one site.DOWNLOAD_DELAYpaces each download slot with ±50 % jitter, and AutoThrottle adjusts that delay to the site’s latency, never belowDOWNLOAD_DELAY.- Four promises are left to you in Scrapy 2.19: it reads Crawl-delay without applying it, reads a robots.txt that answers 503 as no rules, retries a 429 without waiting for its Retry-After, and replays its cache without asking under DummyPolicy.
Terms in this lesson
- AutoThrottle
- A Scrapy extension, off by default, that changes each download slot's delay as a crawl runs: it starts at 5 s and moves halfway towards the site's latency after each response, never below DOWNLOAD_DELAY.
- download slot
- The queue in Scrapy's downloader for the requests of one host, by default. DOWNLOAD_DELAY, its jitter and CONCURRENT_REQUESTS_PER_DOMAIN apply to each slot, so every host is paced on its own.
References
- Scrapy 2.19: Settings (ROBOTSTXT_OBEY, DOWNLOAD_DELAY, DOWNLOAD_DELAY_JITTER) (Scrapy developers)
- Scrapy 2.19: AutoThrottle extension (Scrapy developers)
- Scrapy 2.19: Downloader middleware (RobotsTxtMiddleware, HttpCacheMiddleware, RetryMiddleware) (Scrapy developers)
- Scrapy 2.19: Signals (robots_parsed, request_reached_downloader) (Scrapy developers)
- Scrapy 2.19: Extensions (Scrapy developers)
- Scrapy: Release notes (2.18 and 2.19) (Scrapy developers)
- RFC 9309: Robots Exclusion Protocol (section 2.3.1) (IETF)
- RFC 9110: HTTP Semantics (section 10.2.3, Retry-After) (IETF)
- RFC 9111: HTTP Caching (sections 4.2 and 5.2.2.4) (IETF)
- Twisted: Deferred.asFuture (Twisted Matrix Laboratories)
- Protego (Scrapy project)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress