Collecting web data: API, feed or scraping?

Nimbu Bazaar, the practice shop of this track, lists 240 products. You can collect every name and price with one download, three API requests, ten page requests or 250 requests that crawl every product page, and with each step down that list the site does more work and your code gets more fragile.

The order in which to try those sources runs through this whole track: an official API or export first, then bulk downloads, feeds and sitemaps, data already in the page, the HTML, and a real browser last, with the site’s permission checked at every step. Below you will see what scraping, crawling and browser automation each send over HTTP, run the practice shop inside your own program so that no request leaves it, and learn which answers from a site mean stop.

The best source is usually not the HTML

Suppose you need the name and price of every product in a shop. The visible pages are the obvious place to look, and usually the last one worth trying, because most sites that hold data also offer it in a form made for programs. Try these six sources in order and take the first one the site offers for your data:

1. An official API or export. The site documents it, its fields keep their names from one release to the next, and its terms say how you may use it. Your own data often comes this way: an account page that exports your orders.
2. A bulk download or open data. One file holds everything, usually under a licence that states its conditions, such as naming the source. Wikimedia, for example, publishes full exports of every public wiki and limits its download servers to 3 connections from one address (Wikimedia downloads), so nobody needs to crawl the articles one page at a time.
3. A feed or a sitemap. An RSS or Atom feed lists new items; a sitemap lists a site’s pages, up to 50,000 in one file, often with the date each page last changed (sitemaps.org). Both tell you what to fetch, so you skip what has not changed.
4. Data already in the page. Many pages carry a block of JSON-LDJSON-LD: JSON for Linked Data: a block of machine-readable facts in a page, usually in the schema.org vocabulary, such as a Product whose Offer gives a price and currency. for search engines, such as a schema.org Product whose Offer gives the price and its currency (schema.org). Reading that block is steadier than reading the visible layout.
5. The HTML. When the data is only in the visible page, you parse the HTML. It works, and a redesign can move every field you read.
6. A real browser, last. Only when the data appears after the page’s scripts have run does a program need a browser engine, which loads every script, style sheet and image the page names.

At every step one question comes before any code: does the site allow this source for your purpose? Its robots.txt, its terms, the licence of the data and the law each have a say, and the next lessons of this module read them one by one. When no source on the ladder is both offered and allowed, you stop, ask the owner, or look elsewhere. Walk the ladder for one need, the names and prices of all 240 products for a price-comparison site that earns money, and watch both questions at every step:

The walk ended at the second step. The API was there, but the shop’s terms allow it for non-commercial use only; the download was there too, under a licence that allows any use as long as the source is named. Had the licence said no as well, the walk would have gone on down with the same two questions, is it offered and is it allowed, and ended with a request to the owner if no step had said yes.

Each method sends a different stream of requests

Three words cover most automated collection, and they describe different work:

• Web scrapingweb scraping: Downloading documents with a program and extracting the fields you need from them, such as a name and a price from each row of a product table. downloads documents with a program and extracts the fields you need, such as a name and a price from each row of a table.
• Web crawlingweb crawling: Discovering documents automatically by following links or reading sitemaps, and fetching each new address found. A crawler usually scrapes what it fetches. discovers the documents: it follows links or reads a sitemap and fetches each address it finds. A crawler usually scrapes what it fetches, so the two often run together.
• Browser automationbrowser automation: Driving a real browser engine from a program: the engine loads a page with every file it names and runs its scripts, as a person's browser would. drives a real browser engine from a program. The engine loads the page as a person’s browser would, with every file the page names, and runs its scripts.

Each request is work the site’s servers do for you: they build or read the page and send its bytes. Step through one collection of the shop’s products by each method, and count the requests as they reach the site:

The program below collects the 240 names and prices from the practice shop in five ways, the three methods and two of the ladder’s better sources, and adds up the requests and bytes the shop sends back for each:

"""Five ways to collect the name and price of all 240 products, and what each one asks of the site."""
import json
from html.parser import HTMLParser

import nimbu_lab
from nimbu_lab import fetch

SITE = "https://bazaar.nimbu.test"
BOT = "NimbuLearner/1.0 (+https://example.org/my-bot; you@example.org)"


class Links(HTMLParser):
    """Collects the addresses a page points to: links, and the files a browser would load."""

    def __init__(self):
        super().__init__()
        self.pages, self.files, self.api = [], [], None

    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag == "a":
            self.pages.append(a["href"])
        elif tag in ("script", "img") and "src" in a:
            self.files.append(a["src"])
        elif tag == "link" and a.get("rel") == "stylesheet":
            self.files.append(a["href"])
        elif tag == "table":
            self.api = a.get("data-api")


def next_link(response):
    """The address in a Link: <...>; rel="next" header, or None on the last page."""
    link = response.headers.get("Link", "")
    return link[1:link.index(">")] if 'rel="next"' in link else None


def listing_pages():
    """Follow the catalogue's next links from page 1, yielding each page's address and parsed links."""
    url = "/products/?page=1"
    while url:
        page = fetch(SITE + url, user_agent=BOT)
        links = Links()
        links.feed(page.text)
        yield url, links
        url = next_link(page)


def bulk_file():
    rows = fetch(SITE + "/data/products.csv", user_agent=BOT).text.splitlines()[1:]
    return len(rows)


def api_pages():
    count, url = 0, "/api/v1/products?page=1&per_page=100"
    while url:
        answer = fetch(SITE + url, user_agent=BOT)
        count += len(json.loads(answer.body)["items"])
        url = next_link(answer)
    return count


def html_pages():
    return sum(len(links.pages) for _, links in listing_pages())


def crawl_product_pages():
    count = 0
    for _, links in listing_pages():
        for href in links.pages:
            fetch(SITE + href, user_agent=BOT)
            count += 1
    return count


def browser_like():
    """No browser runs here: this asks for what each listing page tells a browser to load."""
    count = 0
    for _, links in listing_pages():
        for file in links.files:
            fetch(SITE + file, user_agent=BOT)
        api = fetch(SITE + links.api.replace("&amp;", "&"), user_agent=BOT)
        count += len(json.loads(api.body)["items"])
    return count


print(f"{'method':32} {'requests':>8} {'bytes':>9} {'products':>8}")
for name, collect in [("bulk download (one CSV file)", bulk_file), ("API, 100 products a request", api_pages),
                      ("HTML listing pages", html_pages), ("crawl every product page", crawl_product_pages),
                      ("what a browser loads", browser_like)]:
    nimbu_lab.LOG.clear()
    found = collect()
    sent = len(nimbu_lab.LOG)
    print(f"{name:32} {sent:8} {sum(e['bytes'] for e in nimbu_lab.LOG):9,} {found:8}")

method                           requests     bytes products
bulk download (one CSV file)            1     7,230      240
API, 100 products a request             3    14,993      240
HTML listing pages                     10    24,288      240
crawl every product page              250   117,227      240
what a browser loads                   50   121,130      240

The table is the case for the ladder in numbers. The CSV file costs the shop 1 request and 7,230 bytes. The API needs 3 requests, the listing pages 10, and crawling every product page 250 requests and 117,227 bytes, sixteen times the bytes of the file for the same 240 rows. At a polite pace of one request a second, the file takes a second to collect and the crawl more than four minutes. The browser row needs a word of care: no browser runs in this program. It asks for what each listing page tells a browser to load (a style sheet, a script, a logo and the script’s call to the API), 5 requests for every page of 24 products. The practice shop’s files are small stand-ins; on real sites the scripts and images are often far larger than the data they display.

The practice site runs inside your program

Every example in this track collects from Nimbu Bazaar, a made-up shop whose products, prices and pages were generated for practice and describe nothing real. Learning on it means your mistakes, a loop that fetches the same page a thousand times or a missing pause between requests, cost nobody anything.

In the examples the shop runs inside your own program. It is a WSGIWSGI: The Web Server Gateway Interface of PEP 3333: the standard way a Python web server calls an application, a function that takes a dictionary describing a request and returns the response body. application: the standard interface between Python web servers and web applications, defined in PEP 3333, in which the application is a function that receives a dictionary describing one request and returns the body of the response (PEP 3333). The helper fetch() builds that dictionary from an address and calls the function directly, so the request never reaches a network. Follow one request through the two files of the next example:

Here are both files. nimbu_lab.py is the shop; first_request.py asks it for its robots.txt and the first page of products, the way the next lessons will. Run it, then press Edit and change the address:

from nimbu_lab import fetch

BOT = "NimbuLearner/1.0 (+https://example.org/my-bot; you@example.org)"

robots = fetch("https://bazaar.nimbu.test/robots.txt", user_agent=BOT)
print(robots.status, robots.headers["Content-Type"])
print(robots.text)

page = fetch("https://bazaar.nimbu.test/products/?page=1", user_agent=BOT)
print(page.status)
for name in ["Content-Type", "Content-Length", "Link"]:
    print(f"{name}: {page.headers[name]}")
first_row = [line for line in page.text.splitlines() if line.startswith("<tr><td>")][0]
print(first_row)

200 text/plain; charset=utf-8
User-agent: *
Disallow: /account/
Disallow: /people/
Allow: /products/
Sitemap: https://bazaar.nimbu.test/sitemap.xml

200
Content-Type: text/html; charset=utf-8
Content-Length: 2426
Link: </products/?page=2>; rel="next"
<tr><td><a href="/products/NB-0001/">Basmati rice 250 g</a></td><td>₹25.00</td></tr>

"""Nimbu Bazaar, the practice shop of this track. Every product and price in it is invented.

The site is a WSGI application (PEP 3333): a Python function that takes a request and returns a response.
fetch() calls that function inside your own program, so no request travels over a network.
"""
import csv
import io
import json
from urllib.parse import parse_qs, urlsplit
from wsgiref.util import setup_testing_defaults

HOST = "bazaar.nimbu.test"
PER_PAGE = 24
KINDS = ["Basmati rice", "Toor dal", "Green tea", "Jaggery", "Turmeric", "Mustard oil", "Poha", "Rock salt",
         "Cashews", "Cardamom", "Ragi flour", "Ghee", "Honey", "Peanuts", "Coffee", "Raisins", "Sesame seeds",
         "Tamarind", "Cumin", "Black pepper", "Rolled oats", "Cloves", "Fennel", "Moong dal", "Chana dal",
         "Wheat flour", "Semolina", "Coriander", "Red chilli", "Almonds", "Dates", "Coconut oil", "Brown rice",
         "Millet", "Flax seeds", "Cinnamon", "Ginger powder", "Vermicelli", "Masoor dal", "Sago"]
SIZES = [250, 500, 1000, 2000, 5000, 10000]  # grams
ROBOTS = """\
User-agent: *
Disallow: /account/
Disallow: /people/
Allow: /products/
Sitemap: https://bazaar.nimbu.test/sitemap.xml
"""
STATIC = {"/static/site.css": b"body{font:16px sans-serif}" * 40,
          "/static/catalogue.js": b"/* draws the table */" * 300,
          "/static/logo.svg": b"<svg xmlns='http://www.w3.org/2000/svg'/>" * 20}
LOG = []  # one entry per request: what the site saw


def products():
    """The 240 products: 40 kinds in 6 pack sizes, each price in paise (100 paise = 1 rupee)."""
    out = []
    for k, kind in enumerate(KINDS):
        per_kg = 60 + (k * 37) % 900  # rupees a kilogram
        for grams in SIZES:
            size = f"{grams // 1000} kg" if grams >= 1000 else f"{grams} g"
            out.append({"sku": f"NB-{len(out) + 1:04d}", "name": f"{kind} {size}",
                        "paise": per_kg * grams // 10 + 1000})
    return out


def rupees(paise):
    return f"₹{paise // 100:,}.{paise % 100:02d}"


def listing(page):
    """One page of the catalogue as HTML: 24 products, the files a browser loads, a link to the next page."""
    items = products()[(page - 1) * PER_PAGE:page * PER_PAGE]
    rows = "".join(f'<tr><td><a href="/products/{p["sku"]}/">{p["name"]}</a></td><td>{rupees(p["paise"])}</td></tr>\n'
                   for p in items)
    return ("<!doctype html>\n<html lang=\"en\"><head><meta charset=\"utf-8\">"
            f"<title>Products, page {page} · Nimbu Bazaar</title>\n"
            '<link rel="stylesheet" href="/static/site.css"><script src="/static/catalogue.js" defer></script>\n'
            '</head><body><img src="/static/logo.svg" alt="Nimbu Bazaar">\n'
            f'<table id="products" data-api="/api/v1/products?page={page}&amp;per_page={PER_PAGE}">\n'
            f"<tr><th>Product</th><th>Price</th></tr>\n{rows}</table></body></html>\n")


def product_page(p):
    """A product's own page, with its name and price as JSON-LD (schema.org Product and Offer)."""
    data = {"@context": "https://schema.org", "@type": "Product", "sku": p["sku"], "name": p["name"],
            "offers": {"@type": "Offer", "price": f'{p["paise"] / 100:.2f}', "priceCurrency": "INR"}}
    return (f'<!doctype html>\n<html lang="en"><head><meta charset="utf-8"><title>{p["name"]}</title>\n'
            f'<script type="application/ld+json">{json.dumps(data)}</script></head>\n'
            f'<body><h1>{p["name"]}</h1><p class="price">{rupees(p["paise"])}</p></body></html>\n')


def app(environ, start_response):
    """The site: answers one request."""
    path, query = environ["PATH_INFO"], parse_qs(environ.get("QUERY_STRING", ""))
    page = int(query.get("page", ["1"])[0])
    by_sku = {p["sku"]: p for p in products()}
    status, kind, extra, body = "200 OK", "text/html; charset=utf-8", [], b""
    if environ.get("HTTP_HOST") != HOST:
        status, kind, body = "421 Misdirected Request", "text/plain", b"not a Nimbu Bazaar address"
    elif path == "/robots.txt":
        kind, body = "text/plain; charset=utf-8", ROBOTS.encode()
    elif path == "/products/" and 1 <= page <= 10:
        body = listing(page).encode()
        if page < 10:
            extra = [("Link", f'</products/?page={page + 1}>; rel="next"')]
    elif path.startswith("/products/NB-") and path[10:-1] in by_sku:
        body = product_page(by_sku[path[10:-1]]).encode()
    elif path == "/api/v1/products":
        per_page = min(int(query.get("per_page", ["24"])[0]), 100)
        items = products()[(page - 1) * per_page:page * per_page]
        last = (len(products()) + per_page - 1) // per_page
        kind, body = "application/json", json.dumps({"page": page, "pages": last, "items": items}).encode()
        if page < last:
            extra = [("Link", f'</api/v1/products?page={page + 1}&per_page={per_page}>; rel="next"')]
    elif path == "/data/products.csv":
        out = io.StringIO()
        writer = csv.writer(out, lineterminator="\n")
        writer.writerow(["sku", "name", "price_inr"])
        writer.writerows([p["sku"], p["name"], f'{p["paise"] / 100:.2f}'] for p in products())
        kind, body = "text/csv; charset=utf-8", out.getvalue().encode()
    elif path in STATIC:
        kind, body = "application/octet-stream", STATIC[path]
    else:
        status, kind, body = "404 Not Found", "text/plain", b"no such page"
    LOG.append({"path": path + ("?" + environ["QUERY_STRING"] if environ.get("QUERY_STRING") else ""),
                "status": int(status[:3]), "bytes": len(body), "user_agent": environ.get("HTTP_USER_AGENT", "")})
    start_response(status, [("Content-Type", kind), ("Content-Length", str(len(body)))] + extra)
    return [body]


class Response:
    def __init__(self, status, headers, body):
        self.status, self.headers, self.body = status, headers, body
        self.text = body.decode("utf-8")


def fetch(url, user_agent):
    """GET a Nimbu Bazaar address by calling app() directly: nothing leaves your program."""
    parts = urlsplit(url)
    environ = {"REQUEST_METHOD": "GET", "PATH_INFO": parts.path or "/", "QUERY_STRING": parts.query,
               "HTTP_HOST": parts.netloc, "HTTP_USER_AGENT": user_agent, "wsgi.url_scheme": parts.scheme}
    setup_testing_defaults(environ)
    answer = {}

    def start_response(status, headers, exc_info=None):
        answer["status"], answer["headers"] = int(status[:3]), dict(headers)

    body = b"".join(app(environ, start_response))
    return Response(answer["status"], answer["headers"], body)

The program sent two requests and printed what came back: status 200 and the shop’s robots.txt for the first, then the headers of the first listing page. Content-Length says the page is 2,426 bytes, and the Link header points to the next page, which is how a collector learns where to go without guessing addresses. The last line is the first product row exactly as the server sent it, HTML tags and all; reading the fields out of such rows is the work of the parsing module. The user_agent argument names the program and says how to reach you, and goes out as the User-AgentUser-Agent: The HTTP request header that names the software sending the request. An honest collector puts its own name, version and a contact address in it. header of every request. HTTP defines that header to identify the software that sends a request, and asks a robot to send a contact address too, so that whoever runs it can be reached when it causes trouble (RFC 9110, sections 10.1.5 and 10.1.2).

On your own computer the same application can also listen as a small web server on the address 127.0.0.1, which only your own machine can reach. The next lesson sets that up, together with the versions of Python and Node this track uses.

Some answers mean stop

A site answers a collector in more ways than with pages. Some answers ask you to slow down, some refuse, and some ask whether you are a person. Each one has a legitimate next step, and none of them is a puzzle to get around:

The status codes come from the HTTP standards: a 403 means the server understood the request and refuses it, and a client should not repeat it automatically with the same credentials (RFC 9110, section 15.5.4); a 429 says the client sent too many requests in a given time and may carry a Retry-After header that says how long to wait (RFC 6585, section 4). Later lessons handle each answer in code.

A block is information about the site’s wishes. A collector that stops at the first block, logs what it met and lets a person decide what to ask for keeps working with the site instead of against it.

Practise: choose the first source

The exercise gives you the ladder as a list and asks for two things: a function that returns the first source a site both offers and allows, or “stop” when there is none, and your answer for twelve real situations, from an export of your own orders to a forum behind a login. The sample tests explain every answer they reject:

LADDER = ["api", "download", "feed", "page-data", "html", "browser"]


def first_source(offered, allowed):
    """The first source in LADDER that the site both offers and allows, or "stop"."""
    # Replace this: scraping the HTML first is the habit this exercise breaks.
    return "html"


# The source you would use first for each of the twelve needs in the prompt, or "stop".
ANSWERS = {}

from sources import ANSWERS, LADDER, first_source

EXPECTED = {
    1: ("api", "the shop's own export of your orders is an official export: the first source on the ladder"),
    2: ("download", "one openly licensed file comes before reading the timetable off its pages"),
    3: ("feed", "the feed lists the new articles, so you fetch only what is new"),
    4: ("page-data", "the JSON-LD block carries the name, price and currency for programs"),
    5: ("html", "the table exists only in the page, and robots.txt allows it"),
    6: ("browser", "the data appears only after the script runs, and nothing else offers it"),
    7: ("html", "the API is offered but not allowed for you; the pages are allowed for your use"),
    8: ("stop", "robots.txt and the terms both say no, and nothing else offers the profiles"),
    9: ("download", "the API and the pages are not allowed for a use that earns money; the file's licence allows it"),
    10: ("feed", "the sitemap gives each article's last change, so you fetch only what changed"),
    11: ("api", "a documented API with a key and limits is the first source to use"),
    12: ("stop", "the terms forbid automated access to the logged-in pages, and there is no API"),
}


def test_takes_the_first_source_that_is_offered_and_allowed():
    """returns the first source of LADDER in both sets"""
    assert first_source({"api", "html"}, {"api", "html"}) == "api"
    assert first_source({"html", "download"}, {"download", "html", "browser"}) == "download"
    assert first_source({"browser", "feed"}, {"browser", "feed"}) == "feed"


def test_skips_a_source_that_is_offered_but_not_allowed():
    """an offered source the site does not allow is never the answer"""
    assert first_source({"api", "html"}, {"html"}) == "html"
    assert first_source({"api", "download", "html"}, {"download"}) == "download"


def test_skips_a_source_that_is_allowed_but_not_offered():
    """a source the site allows but does not offer for this data is never the answer"""
    assert first_source({"html"}, {"api", "download", "html"}) == "html"


def test_stops_when_no_source_is_both():
    """returns "stop" when no source is both offered and allowed"""
    assert first_source({"api", "html"}, {"download"}) == "stop"
    assert first_source(set(), set(LADDER)) == "stop"
    assert first_source(set(LADDER), set()) == "stop"


def test_answers_for_the_twelve_needs():
    """the first source for each need of the prompt"""
    assert sorted(ANSWERS) == list(range(1, 13)), "give an answer for each need, 1 to 12"
    for need, (want, why) in EXPECTED.items():
        assert ANSWERS[need] == want, f"need {need}: {want}, because {why} (you wrote {ANSWERS[need]!r})"

After the exercise, the self-check goes over the ladder, the three methods, the lab shop and the answers that mean stop. It picks its first question from whichever of the four your checkpoint answers found hardest:

Interview questions

Warm-up (fresher): What is the difference between web scraping and web crawling?

Web scraping extracts data from documents a program has fetched: it reads a page, a feed or a file and pulls out fields such as a product’s name and price. Web crawling discovers documents: it starts from some addresses, follows links or reads sitemaps, and fetches each new address it finds, usually within rules such as robots.txt. A crawler usually scrapes what it fetches, so the two often run together, but either can run alone: a scraper can work on ten pages you listed by hand, and a web archive’s crawler stores each page whole without extracting any field. Both send requests that cost the site work, so both follow the same permissions.

Two harder questions grow out of this lesson: a partner that wants its own prices collected, and a site owner who writes to say your collector is slowing the site down. The question page of this module gives a first step and the follow-ups for both:

Key takeaways

• Try the sources in order, an official API or export, a bulk download, a feed or sitemap, data in the page, the HTML and a real browser last, and take the first one the site both offers and allows for your purpose.
• Scraping extracts fields from documents, crawling discovers documents by following links or sitemaps, and browser automation drives a real browser engine that loads every file a page names.
• In the practice shop the same 240 products cost 1 request as a file, 3 through the API, 10 as listing pages and 250 as a crawl of every product page.
• The practice shop is a WSGI application that the examples call inside their own program, so no request reaches a network while you learn.
• A disallow rule, a 403, a 429, a CAPTCHA or terms that say no each mean stop or slow down, and the legitimate next step is to wait, ask the owner or use another source.