Web Scraping & Browser Automation (Python and JavaScript) Module 3 – Parsing HTML in Python and JavaScript

HTML, the DOM and DevTools for scrapers

Read a page as a tree of elements, text and comments, compare the HTML a server sent with the DOM a browser builds, and see why copied selectors fail.

Beginner
20 minutes
Examples run with Python 3.14.8, Pyodide 314.0.7, Node.js 24.21.0, quickjs 0.32.0, beautifulsoup4 4.15.0, html5lib 1.1 and soupsieve 2.10
By MySmartCoPilot

Free: the whole lesson. With a pass: the model answers to its 2 interview questions.

What you will learn

Read an HTML document as a tree of element, text and comment nodes, with attributes held by their elements
Predict how a browser repairs markup, such as the tbody element it inserts into a table
Compare the page source with the live DOM, and name the DevTools view that shows each
Explain why a selector copied from DevTools can match nothing in the HTML a script downloads

Before you start

HTTP for scrapers: requests, responses, headers

What you’ll learn · Before you start

What you will learn

Read an HTML document as a tree of element, text and comment nodes, with attributes held by their elements
Predict how a browser repairs markup, such as the tbody element it inserts into a table
Compare the page source with the live DOM, and name the DevTools view that shows each
Explain why a selector copied from DevTools can match nothing in the HTML a script downloads

Before you start

HTTP for scrapers: requests, responses, headers

On this page

Page 1 of the practice shop’s catalogue arrives as 2,426 bytes of HTML holding 25 table rows, and a browser turns those bytes into a tree before it draws a single row. Copy a selector for the first product from the browser’s developer tools, run it on the same bytes with Python’s own parser, and it finds 1 element in the browser’s tree and 0 in yours.

Here you read a page the way a parser does: as a tree of elements, text and comments built from the markup in order. You will see where a browser’s tree differs from the HTML the server sent, which view of the browser’s DevTools shows which of the two, and how to write a selector that finds the same element in both.

A page is a tree of elements, text and comments

Here is one product card from the shop, as its HTML is written:

card.html: one product card HTML · tree/card.html

<li class="product" data-sku="NB-0013">
  <a href="/products/NB-0013/">Green tea 250 g</a>
  <span class="price">₹43.50</span>
  <!-- stock is filled in by catalogue.js -->
  <span class="note">Tea &amp; coffee&nbsp;shelf</span>
</li>

A parser reads markup from the first character to the last and builds a tree, the DOM: Document Object Model: the tree of nodes that a browser or a parser builds from a page, with elements, text and comments as nodes and attributes stored on their elements. Scripts read and change the page through it.. Each tag pair becomes an element node with a name, such as li, and its attributes. The characters between tags become text node: A node of the DOM that holds a run of characters between two tags, such as a product name, or the spaces and line break between two elements. Entities such as & are already decoded in it., the spaces and line breaks between two tags included, and the comment becomes a comment node. Attributes are not children: they belong to their element, and the DOM Standard says that attribute nodes never have a parent or any children of their own.

Python’s own html.parser does not build that tree itself. It reports what it reads, one event at a time: a start tag, some text, an end tag. A tree builder turns those events into nodes with a stack of open elements: a start tag adds a child to the element on top of the stack and opens it, text goes into the open element, and an end tag closes it again. Step through the card:

Interactive · message by message A parser's events become a tree, one node at a time Step through html.parser reading the product card, and watch the stack of open elements decide where each node goes.

The drawing shows the whole scene; its 9 steps are listed below.

Every step as text

A start tag. The parser reads <li class="product" data-sku="NB-0013"> and reports a start tag with two attributes. The builder makes an element node, stores the attributes on it and pushes it: li is now open.
Whitespace is text too. Next come a line break and two spaces. The parser reports them as data, so the builder adds a whitespace text node to the element that is open, li. Nothing on the screen shows it, but it is a node.
A child element. The <a href> start tag makes a second element node, a child of li because li is the open element. The builder pushes it, so the next text lands inside the link.
Text inside the link. The data Green tea 250 g becomes a text node inside a. This is the text a scraper usually wants: it belongs to the a element, not to li directly.
An end tag closes. The </a> end tag pops a off the stack. li is the open element again, so the next node becomes a sibling of a, not its child.
The price, the same way. The price span opens, its text becomes a text node inside it, and its end tag closes it. Two more whitespace text nodes sit between the elements.
A comment is its own node. The comment about catalogue.js becomes a comment node in li. It is part of the tree, but text content and get_text() leave it out.
Entities arrive decoded. In the note, the parser has already turned & into & and   into a no-break space, U+00A0, before it reports the data. The text node holds characters, not the entity names.
The tree is complete. The </li> end tag empties the stack, and the line break that ends the file becomes one last whitespace text node at the top. The file's tree is 4 elements, 3 text nodes with letters, 6 whitespace text nodes and 1 comment: 14 nodes.

The program below is that builder. It skips the stack for the void element: An HTML element that has only a start tag and never any content, such as img, meta, link, br and input. A parser closes it at once, so it never stays open around later elements., such as img, meta and link, which the HTML Standard says “only have a start tag”, and prints every node with its depth:

The card's tree, node by node

tree/tree_walk.py

"""Build the tree of one product card with Python's own html.parser, then print every node with its depth."""
from html.parser import HTMLParser
from pathlib import Path

# Elements that never have an end tag or children (HTML Standard, "void elements").
VOID = {"area", "base", "br", "col", "embed", "hr", "img", "input", "link", "meta", "source", "track", "wbr"}


class Node:
    def __init__(self, kind, name="", attrs=(), text=""):
        self.kind, self.name, self.attrs, self.text = kind, name, dict(attrs), text
        self.children = []


class TreeBuilder(HTMLParser):
    """html.parser reports start tags, text and end tags in order; a stack of open elements makes them a tree."""

    def __init__(self):
        super().__init__()  # convert_charrefs=True: &amp; and &nbsp; arrive as the characters they stand for
        self.document = Node("document")
        self.open = [self.document]

    def handle_starttag(self, tag, attrs):
        node = Node("element", tag, attrs)
        self.open[-1].children.append(node)
        if tag not in VOID:
            self.open.append(node)

    def handle_endtag(self, tag):
        if any(n.name == tag for n in self.open):
            while self.open.pop().name != tag:
                pass

    def handle_data(self, data):
        self.open[-1].children.append(Node("text", text=data))

    def handle_comment(self, data):
        self.open[-1].children.append(Node("comment", text=data))


def show(node, depth=0, counts=None):
    for child in node.children:
        pad = "  " * depth
        if child.kind == "element":
            attrs = "".join(f' {k}="{v}"' for k, v in child.attrs.items())
            print(f"{pad}element <{child.name}>{attrs}")
            show(child, depth + 1, counts)
            kind = "element"
        elif child.kind == "text":
            kind = "whitespace text" if not child.text.strip() else "text"
            print(f"{pad}{kind} {child.text!r}")
        else:
            kind = "comment"
            print(f"{pad}comment {child.text!r}")
        counts[kind] = counts.get(kind, 0) + 1
    return counts


def text_content(node):
    """All the text below a node, joined, comments left out: what textContent and get_text() return."""
    if node.kind == "text":
        return node.text
    return "".join(text_content(c) for c in node.children if c.kind != "comment")


builder = TreeBuilder()
builder.feed(Path("card.html").read_text(encoding="utf-8"))
builder.close()
counts = show(builder.document, counts={})
print()
print("nodes:", ", ".join(f"{counts.get(k, 0)} {k}" for k in ("element", "text", "whitespace text", "comment")))
card = builder.document.children[0]
print("attributes of <li>, not children:", card.attrs)
print("text content of <li>:", repr(text_content(card)))

Output

element <li> class="product" data-sku="NB-0013"
  whitespace text '\n  '
  element <a> href="/products/NB-0013/"
    text 'Green tea 250 g'
  whitespace text '\n  '
  element <span> class="price"
    text '₹43.50'
  whitespace text '\n  '
  comment ' stock is filled in by catalogue.js '
  whitespace text '\n  '
  element <span> class="note"
    text 'Tea & coffee\xa0shelf'
  whitespace text '\n'
whitespace text '\n'

nodes: 4 element, 3 text, 6 whitespace text, 1 comment
attributes of <li>, not children: {'class': 'product', 'data-sku': 'NB-0013'}
text content of <li>: '\n  Green tea 250 g\n  ₹43.50\n  \n  Tea & coffee\xa0shelf\n'

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 tree_walk.py

tree/card.html

<li class="product" data-sku="NB-0013">
  <a href="/products/NB-0013/">Green tea 250 g</a>
  <span class="price">₹43.50</span>
  <!-- stock is filled in by catalogue.js -->
  <span class="note">Tea &amp; coffee&nbsp;shelf</span>
</li>

Four elements, three text nodes with letters in them, six whitespace text nodes and one comment: fourteen nodes for a file that shows three lines of text. Two details matter for scraping. The entities are already decoded, so the note holds & where the file says &, and a no-break space, \xa0, where it says  : a test such as note == "Tea & coffee shelf" is false. And the text content, which a browser’s textContent and Beautiful Soup’s get_text() give you, joins every text node below an element, whitespace included, and leaves the comment out.

Flatten the text yourself

Press Edit and change the last line to print(repr(" ".join(text_content(card).split()))), then run it again. The card’s text becomes 'Green tea 250 g ₹43.50 Tea & coffee shelf': split() breaks at every run of whitespace, the no-break space included, and the join puts back one plain space.

HTML Formatter Indent a page that arrives as one long line, so its elements read as a tree before you write a selector.

Checkpoint

Check: Elements, text and comments as a tree

Question 1 of 2 How many nodes does the tree builder of tree_walk.py make for <p class="price">₹43.50 <small>incl. tax</small></p>, counting elements and text nodes?
Type a number.
Your answer nodes

A hint

Count the elements first, then each run of text between two tags.

Show the answer: Question 1 of 2

Answer: 4 nodes

Two elements, p and small, and two text nodes: "₹43.50 " inside p, before small, and "incl. tax" inside small. The class attribute is not a node of its own in the tree: it is stored on the p element.
Question 2 of 2 In the card's tree, where does data-sku="NB-0013" live?
Choose one answer.

In a child node of li, before the first whitespace text node Nowhere, because parsers drop data-* attributes On the li element, as one of its attributes In a text node inside li

Show the answer: Question 2 of 2

Answer: On the li element, as one of its attributes

An element node carries its attributes with it, and the tree's children are elements, text and comments. In the card, li holds class and data-sku as attributes, and its children start with a whitespace text node.

A browser repairs the markup before you see it

Look at the catalogue table in the HTML the shop sends: rows follow the <table> tag directly, and no <tbody> tag appears anywhere. That is valid HTML. The HTML Standard lets a tbody element’s start tag be left out when its first child is a tr, so the element is there even though its tags are not.

A browser builds its tree by the HTML Standard’s parsing rules, which say what to do with every piece of markup, valid or broken. One of them matters to every scraper. When a tr start tag arrives directly inside a table, the parser must first create a tbody element, with no attributes (section 13.2.6.4.9), and then place the row inside it. The same rules add html, head and body elements to a page that leaves them out. Python’s html.parser applies none of these rules: it reports the tags as they are written. Follow the first row through both:

Interactive · step by step The same table built as written and as a browser builds it Step through the first row of the catalogue table in both trees, and watch where the browser's tbody comes from.

The drawing shows the whole scene; its 6 steps are listed below.

Every step as text

The table opens. Both parsers read <table id="products" data-api="…"> and make a table element with its two attributes. So far the two trees are the same.
html.parser takes the tr. Rows follow: first the header row, then the first product. html.parser reports each <tr> as written, so every row becomes a child of the table: the left tree reads table > tr.
A browser inserts tbody. A browser builds the tree by the HTML Standard. The first tr start tag that arrives directly in a table, the header row, makes it insert a tbody element first, with no attributes, though the server never sent one.
Then the row goes inside it. The parser then reprocesses that tr inside tbody, and every later row joins it there, the first product too. The right tree reads table > tbody > tr.
Cells and links follow. The td and a elements follow the same way in both trees. Only the step between table and tr differs.
25 rows, two answers. All 25 rows end up under tbody in the browser tree and directly under table in html.parser's, so table > tbody > tr finds 25 rows in one tree and 0 in the other, as source_vs_dom.py prints.

You do not need a browser to see the browser’s tree. html5lib is a Python parser “designed to conform to the WHATWG HTML specification”, which browsers implement. Beautiful Soup 4.15.0 can run either parser on the same bytes, and the program below counts what each tree holds:

The same bytes as html.parser and html5lib build them

dom/source_vs_dom.py

"""The same bytes, two trees: Python's html.parser keeps the markup as written, html5lib builds the tree the way the
HTML Standard tells browsers to (Beautiful Soup 4.15.0 runs both)."""
from pathlib import Path

from bs4 import BeautifulSoup

listing = Path("listing.html").read_text(encoding="utf-8")
card = Path("card.html").read_text(encoding="utf-8")

print("page 1 of the catalogue (no <tbody> in the HTML):")
for parser in ("html.parser", "html5lib"):
    soup = BeautifulSoup(listing, parser)
    print(f"  {parser:<11} <tbody> elements: {len(soup.find_all('tbody'))}"
          f" | table > tbody > tr: {len(soup.select('table > tbody > tr')):>2}"
          f" | table > tr: {len(soup.select('table > tr')):>2}")

print("one product card (no <html>, <head> or <body> in the HTML):")
for parser in ("html.parser", "html5lib"):
    soup = BeautifulSoup(card, parser)
    price = soup.select_one(".price")
    path = [p.name for p in reversed(list(price.parents)) if p.name != "[document]"] + [price.name]
    print(f"  {parser:<11} path to the price: {' > '.join(path)}")

Output

page 1 of the catalogue (no <tbody> in the HTML):
  html.parser <tbody> elements: 0 | table > tbody > tr:  0 | table > tr: 25
  html5lib    <tbody> elements: 1 | table > tbody > tr: 25 | table > tr:  0
one product card (no <html>, <head> or <body> in the HTML):
  html.parser path to the price: li > span
  html5lib    path to the price: html > body > li > span

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 source_vs_dom.py

listing.html (page 1) · dom/listing.html

<!doctype html>
<html lang="en"><head><meta charset="utf-8"><title>Products, page 1 · Nimbu Bazaar</title>
<link rel="stylesheet" href="/static/site.css"><script src="/static/catalogue.js" defer></script>
</head><body><img src="/static/logo.svg" alt="Nimbu Bazaar">
<table id="products" data-api="/api/v1/products?page=1&amp;per_page=24">
<tr><th>Product</th><th>Price</th></tr>
<tr><td><a href="/products/NB-0001/">Basmati rice 250 g</a></td><td>₹25.00</td></tr>
<tr><td><a href="/products/NB-0002/">Basmati rice 500 g</a></td><td>₹40.00</td></tr>
<tr><td><a href="/products/NB-0003/">Basmati rice 1 kg</a></td><td>₹70.00</td></tr>
<tr><td><a href="/products/NB-0004/">Basmati rice 2 kg</a></td><td>₹130.00</td></tr>
<tr><td><a href="/products/NB-0005/">Basmati rice 5 kg</a></td><td>₹310.00</td></tr>
<tr><td><a href="/products/NB-0006/">Basmati rice 10 kg</a></td><td>₹610.00</td></tr>
<tr><td><a href="/products/NB-0007/">Toor dal 250 g</a></td><td>₹34.25</td></tr>
<tr><td><a href="/products/NB-0008/">Toor dal 500 g</a></td><td>₹58.50</td></tr>
<tr><td><a href="/products/NB-0009/">Toor dal 1 kg</a></td><td>₹107.00</td></tr>
<tr><td><a href="/products/NB-0010/">Toor dal 2 kg</a></td><td>₹204.00</td></tr>
<tr><td><a href="/products/NB-0011/">Toor dal 5 kg</a></td><td>₹495.00</td></tr>
<tr><td><a href="/products/NB-0012/">Toor dal 10 kg</a></td><td>₹980.00</td></tr>
<tr><td><a href="/products/NB-0013/">Green tea 250 g</a></td><td>₹43.50</td></tr>
<tr><td><a href="/products/NB-0014/">Green tea 500 g</a></td><td>₹77.00</td></tr>
<tr><td><a href="/products/NB-0015/">Green tea 1 kg</a></td><td>₹144.00</td></tr>
<tr><td><a href="/products/NB-0016/">Green tea 2 kg</a></td><td>₹278.00</td></tr>
<tr><td><a href="/products/NB-0017/">Green tea 5 kg</a></td><td>₹680.00</td></tr>
<tr><td><a href="/products/NB-0018/">Green tea 10 kg</a></td><td>₹1,350.00</td></tr>
<tr><td><a href="/products/NB-0019/">Jaggery 250 g</a></td><td>₹52.75</td></tr>
<tr><td><a href="/products/NB-0020/">Jaggery 500 g</a></td><td>₹95.50</td></tr>
<tr><td><a href="/products/NB-0021/">Jaggery 1 kg</a></td><td>₹181.00</td></tr>
<tr><td><a href="/products/NB-0022/">Jaggery 2 kg</a></td><td>₹352.00</td></tr>
<tr><td><a href="/products/NB-0023/">Jaggery 5 kg</a></td><td>₹865.00</td></tr>
<tr><td><a href="/products/NB-0024/">Jaggery 10 kg</a></td><td>₹1,720.00</td></tr>
</table></body></html>

dom/card.html

<li class="product" data-sku="NB-0013">
  <a href="/products/NB-0013/">Green tea 250 g</a>
  <span class="price">₹43.50</span>
  <!-- stock is filled in by catalogue.js -->
  <span class="note">Tea &amp; coffee&nbsp;shelf</span>
</li>

In html5lib’s tree all 25 rows sit inside one tbody, so table > tbody > tr finds 25 and table > tr finds none; in html.parser’s tree it is the other way round. The card, which has no html or body tags, gains both in the browser’s tree, so the path to its price is html > body > li > span there and li > span in html.parser’s. The examples that use Beautiful Soup and html5lib need the track’s packages, which the lab kit installs in a virtual environment.

Writing tbody because the browser shows it

A selector such as #products > tbody > tr copies a step that only the browser added. It finds every row in the browser and none in a script that parses the page with html.parser. Write #products tr, with a space, which matches a row at any depth below the table, so it works in both trees.

HTML Validator Check a page for the markup errors a browser will repair, before your parser meets them.

Checkpoint

Check: How a browser repairs markup

Question 1 of 2 Which selector finds every row of the catalogue table, the header row included, both in html.parser's tree and in a browser's?
Choose one answer.

#products > tr #products tr #products > tbody > tr table > tbody > tr:nth-child(2)

Show the answer: Question 1 of 2

Answer: #products tr

The trees differ only between table and tr. A descendant selector, written with a space, does not care whether the browser's tbody is there, so it gives the same 25 rows from the bytes and from the DOM.
Question 2 of 2 Another shop writes its table with the tags spelled out, <table><tbody><tr>…</tr></tbody></table>. What does table > tbody > tr find with html.parser and with html5lib?
Choose one answer.

Every row with html.parser, but none with html5lib, which adds a second tbody Every row with html5lib, but none with html.parser No rows in either tree, because tbody is optional Every row in both trees

Show the answer: Question 2 of 2

Answer: Every row in both trees

The inserted tbody comes from markup that leaves the tag out. A page that writes the tag gets the same element in both trees, so the difference this lesson measured exists only on pages like the shop's catalogue.

Scripts can change the DOM after the HTML arrives

The catalogue page asks for more than its own HTML. The program below reads it with html.parser, which runs nothing, and lists what a browser would do next:

What the page asks a browser to load and run

hints/data_hints.py

"""What page 1 of the catalogue asks a browser to do beyond its own HTML: files to load, a script to run, and
addresses kept in data-* attributes. Read with Python's own html.parser, so no script runs."""
from html.parser import HTMLParser
from pathlib import Path


class Hints(HTMLParser):
    def __init__(self):
        super().__init__()
        self.loads, self.data_attributes, self.rows = [], [], 0

    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag == "script" and a.get("src"):
            self.loads.append(("script", a["src"], "runs after parsing" if "defer" in a else "runs at once"))
        elif tag == "link" and a.get("rel") == "stylesheet":
            self.loads.append(("style sheet", a["href"], ""))
        elif tag == "img":
            self.loads.append(("image", a["src"], ""))
        for name, value in attrs:
            if name.startswith("data-"):
                where = f"<{tag} id={a['id']}>" if a.get("id") else f"<{tag}>"
                self.data_attributes.append((where, name, value))
        if tag == "tr":
            self.rows += 1


page = Hints()
page.feed(Path("listing.html").read_text(encoding="utf-8"))
page.close()
print("files the browser loads next:")
for kind, address, note in page.loads:
    print(f"  {kind:<12} {address}  {note}".rstrip())
print("data-* attributes:")
for where, name, value in page.data_attributes:
    print(f"  {where} {name} = {value}")
print("table rows in the HTML as sent:", page.rows)

Output

files the browser loads next:
  style sheet  /static/site.css
  script       /static/catalogue.js  runs after parsing
  image        /static/logo.svg
data-* attributes:
  <table id=products> data-api = /api/v1/products?page=1&per_page=24
table rows in the HTML as sent: 25

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 data_hints.py

listing.html (page 1) · hints/listing.html

<!doctype html>
<html lang="en"><head><meta charset="utf-8"><title>Products, page 1 · Nimbu Bazaar</title>
<link rel="stylesheet" href="/static/site.css"><script src="/static/catalogue.js" defer></script>
</head><body><img src="/static/logo.svg" alt="Nimbu Bazaar">
<table id="products" data-api="/api/v1/products?page=1&amp;per_page=24">
<tr><th>Product</th><th>Price</th></tr>
<tr><td><a href="/products/NB-0001/">Basmati rice 250 g</a></td><td>₹25.00</td></tr>
<tr><td><a href="/products/NB-0002/">Basmati rice 500 g</a></td><td>₹40.00</td></tr>
<tr><td><a href="/products/NB-0003/">Basmati rice 1 kg</a></td><td>₹70.00</td></tr>
<tr><td><a href="/products/NB-0004/">Basmati rice 2 kg</a></td><td>₹130.00</td></tr>
<tr><td><a href="/products/NB-0005/">Basmati rice 5 kg</a></td><td>₹310.00</td></tr>
<tr><td><a href="/products/NB-0006/">Basmati rice 10 kg</a></td><td>₹610.00</td></tr>
<tr><td><a href="/products/NB-0007/">Toor dal 250 g</a></td><td>₹34.25</td></tr>
<tr><td><a href="/products/NB-0008/">Toor dal 500 g</a></td><td>₹58.50</td></tr>
<tr><td><a href="/products/NB-0009/">Toor dal 1 kg</a></td><td>₹107.00</td></tr>
<tr><td><a href="/products/NB-0010/">Toor dal 2 kg</a></td><td>₹204.00</td></tr>
<tr><td><a href="/products/NB-0011/">Toor dal 5 kg</a></td><td>₹495.00</td></tr>
<tr><td><a href="/products/NB-0012/">Toor dal 10 kg</a></td><td>₹980.00</td></tr>
<tr><td><a href="/products/NB-0013/">Green tea 250 g</a></td><td>₹43.50</td></tr>
<tr><td><a href="/products/NB-0014/">Green tea 500 g</a></td><td>₹77.00</td></tr>
<tr><td><a href="/products/NB-0015/">Green tea 1 kg</a></td><td>₹144.00</td></tr>
<tr><td><a href="/products/NB-0016/">Green tea 2 kg</a></td><td>₹278.00</td></tr>
<tr><td><a href="/products/NB-0017/">Green tea 5 kg</a></td><td>₹680.00</td></tr>
<tr><td><a href="/products/NB-0018/">Green tea 10 kg</a></td><td>₹1,350.00</td></tr>
<tr><td><a href="/products/NB-0019/">Jaggery 250 g</a></td><td>₹52.75</td></tr>
<tr><td><a href="/products/NB-0020/">Jaggery 500 g</a></td><td>₹95.50</td></tr>
<tr><td><a href="/products/NB-0021/">Jaggery 1 kg</a></td><td>₹181.00</td></tr>
<tr><td><a href="/products/NB-0022/">Jaggery 2 kg</a></td><td>₹352.00</td></tr>
<tr><td><a href="/products/NB-0023/">Jaggery 5 kg</a></td><td>₹865.00</td></tr>
<tr><td><a href="/products/NB-0024/">Jaggery 10 kg</a></td><td>₹1,720.00</td></tr>
</table></body></html>

A browser fetches a style sheet, an image and a script, catalogue.js. The script carries the defer attribute, which the HTML Standard defines as fetching it in parallel and running it “when the page has finished parsing”. From then on it may add, remove or change any node. The Chrome DevTools documentation puts the difference in a few words: the HTML is the initial content, and the DOM “represents active, current page content”. The table also carries a data-api attribute with a JSON address (& already decoded to &), a common sign that a script fetches the products and redraws the rows.

A collector that downloads the page holds the bytes and nothing else: the 25 rows as sent, before any script. Follow the page through a browser, and see where a collector’s copy branches off:

Interactive · step by step From the HTML a server sends to the DOM on screen Follow page 1 of the catalogue down the browser's path, and see where a collector's copy branches off.

The drawing shows the whole scene; its 7 steps are listed below.

Every step as text

Bytes arrive. The server answers with 2,426 bytes of HTML that hold 25 table rows. This is all a collector that downloads the page ever receives, and what View source shows in a browser.
A collector stops here. A script that downloads the page keeps exactly these bytes. It may parse them, but nothing in them runs: no style sheet, no image and no script is fetched.
A browser builds the DOM. A browser parses the bytes by the HTML Standard and repairs them as it goes. Here it inserts a tbody into the table, so its tree already differs from the bytes.
Three more requests. The HTML names three files: a style sheet, a script and an image. The browser requests each of them; the Network panel lists every one with its answer.
The script runs after parsing. catalogue.js carries the defer attribute, so it runs once the document has been parsed. A script can add, remove or change any node of the DOM.
It may fetch the data. The table's data-api attribute names a JSON address. A script can fetch it and redraw the rows; whether this one does is something the Network panel shows, under Fetch/XHR.
What Elements shows. The Elements panel shows the DOM as it is now, after the repairs and after every script. The DevTools documentation calls the DOM the active, current page content; your collector's copy never sees this version.

So when a page in your browser shows data that your collector cannot find, compare two things first: the page source, which is what your collector downloads, and the live DOM. If the data is only in the live DOM, a script put it there, and the Network panel shows the request it came from. Fetching that address directly, when the site allows it, is the subject of a later module.

Checkpoint

Check: Scripts and the live DOM

Question 1 of 2 The Elements panel shows 48 products on a page, but View source and the HTML your collector downloads hold 12. Where are the other 36 most likely from, and where do you look first?
Choose one answer.

The browser's parser repaired the markup and created them The Elements panel shows an old copy from the browser's cache The download was cut short, so fetch the page again A script added them after parsing; look in the Network panel under Fetch/XHR for the request that returned them

Show the answer: Question 1 of 2

Answer: A script added them after parsing; look in the Network panel under Fetch/XHR for the request that returned them

Compare the source with the live DOM first. Data that is only in the live DOM was put there by a script after the HTML arrived, usually from a JSON request that the Network panel shows under Fetch/XHR.
Question 2 of 2 The catalogue page loads <script src="/static/catalogue.js" defer>. When does a browser run that script?
Choose one answer.

As soon as the parser reaches the script tag, before the table is read When the page has finished parsing Only when the user scrolls to the table Never, because the page also has a data-api attribute

Show the answer: Question 2 of 2

Answer: When the page has finished parsing

A deferred script runs after the parser has built the whole DOM, so it can change any node of the page, including the table, before the user sees the final version.

DevTools answers four questions in three places

DevTools: The developer tools built into a browser: the Elements panel shows the live DOM, the Network panel every request the page made, and the Console runs code against the page. answers a scraper’s questions in different places, and each place shows a different version of the page:

Your question	Where to look	What it shows
What did the server send?	View source: Ctrl+U, or Command+Option+U on a Mac	The bytes your collector downloads
Which tree is on screen?	The Elements panel	The live DOM, repaired and changed by scripts
Where did the data come from?	The Network panel, filtered to Fetch/XHR	Each request the page made, with its response
Does my selector match?	Ctrl+F (Command+F) in the Elements panel	Matches of a CSS selector or XPath, in the live DOM only

The last row hides a trap. The Elements panel’s search and its Copy menu both work on the live DOM. In the node’s context menu, Copy, then Copy selector writes a CSS selector: A pattern that picks elements from a tree by tag, class, id, attribute, position or relation, such as #products a[href^="/products/"]. Browsers, Beautiful Soup and lxml all accept them. More in “CSS selectors that survive redesigns” for the element, and DevTools builds it with a function called cssPath, whose source is public: it starts at the element, adds one step for each parent joined with >, gives an element a position such as :nth-child(2) when a sibling has the same tag and no class tells them apart, and stops at the first parent with an id. Follow the walk for the first product link:

Interactive · stage by stage How Copy selector walks up from the first product link Step from the link up through its parents, and see why each step of the copied selector looks the way it does.

The drawing shows the whole scene; its 5 steps are listed below.

Every step as text

Start at the element. You right-click the first product link and choose Copy, then Copy selector. DevTools starts at that a element. Its cell holds no other a, so the step is just a.
Siblings need a position. Its parent td has a sibling td, the price cell, and neither has a class that tells them apart. So the step becomes td:nth-child(1), the first child of its row.
The header row counts. The row has 24 sibling rows and no classes. It is the second child of its parent, because the header row comes first: tr:nth-child(2).
The inserted tbody. The parent of the row is the tbody the browser inserted. It has no sibling of the same tag, so the step is plain tbody, a tag the server never sent.
An id ends the walk. The table has the id products, so DevTools writes #products and stops. Joined with > from the top, the copied selector is #products > tbody > tr:nth-child(2) > td:nth-child(1) > a.

devtools_path.py makes the same walk in Python, on html5lib’s tree, so the copied selector is the one the Elements panel would give you. The program then runs each copied selector on both trees, and one written by hand:

Copied selectors against the HTML a script parses

dom/copied_selector.py

"""Copy a selector the way DevTools does, from the browser's tree, then run it on the HTML the server sent."""
from pathlib import Path

from bs4 import BeautifulSoup

from devtools_path import copy_selector

listing = Path("listing.html").read_text(encoding="utf-8")
browser = BeautifulSoup(listing, "html5lib")  # the tree a browser builds: the Elements panel shows this one
script = BeautifulSoup(listing, "html.parser")  # the tree a script builds from the same bytes

first_name = browser.find("a", string="Basmati rice 250 g")
first_price = first_name.find_parent("tr").find_all("td")[1]
for label, tag in (("first product's link", first_name), ("first product's price", first_price)):
    selector = copy_selector(tag)
    print(f"Copy selector, {label}: {selector}")
    print(f"  matches in the browser's tree: {len(browser.select(selector))}"
          f" | in html.parser's tree: {len(script.select(selector))}")

sturdy = '#products a[href^="/products/NB-"]'
print(f"written by hand: {sturdy}")
print(f"  matches in the browser's tree: {len(browser.select(sturdy))}"
      f" | in html.parser's tree: {len(script.select(sturdy))}")

Output

Copy selector, first product's link: #products > tbody > tr:nth-child(2) > td:nth-child(1) > a
  matches in the browser's tree: 1 | in html.parser's tree: 0
Copy selector, first product's price: #products > tbody > tr:nth-child(2) > td:nth-child(2)
  matches in the browser's tree: 1 | in html.parser's tree: 0
written by hand: #products a[href^="/products/NB-"]
  matches in the browser's tree: 24 | in html.parser's tree: 24

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 copied_selector.py

dom/devtools_path.py

"""The selector that Chrome DevTools copies with Copy > Copy selector, for a Beautiful Soup tag.

DevTools builds it in front_end/panels/elements/DOMPath.ts (cssPath with optimized=true): one step per element,
from the element up through its parents, joined with " > ". A step is the tag name, plus :nth-child(n) when a
sibling has the same tag and the classes do not tell them apart, or the classes when they do. The walk stops at
the first element with an id (the step is then #id) or at body, head or html. This is the same walk in Python.
"""

from bs4 import Tag
from soupsieve import escape


def copy_selector(tag):
    steps = []
    node = tag
    while isinstance(node, Tag) and node.name != "[document]":
        step, stop = _step(node, node is tag)
        steps.append(step)
        if stop:
            break
        node = node.parent
    return " > ".join(reversed(steps))


def _step(node, is_target):
    if node.get("id"):
        return "#" + escape(node["id"]), True
    if node.name in ("body", "head", "html"):
        return node.name, True
    parent = node.parent
    if parent is None or parent.name == "[document]":
        return node.name, True
    own = list(node.get("class", []))
    needs_classes = needs_nth = False
    own_index = -1
    for index, sibling in enumerate(c for c in parent.children if isinstance(c, Tag)):
        if sibling is node:
            own_index = index
            continue
        if needs_nth or sibling.name != node.name:
            continue
        needs_classes = True
        left = set(own)
        if not left:
            needs_nth = True
            continue
        for name in sibling.get("class", []):
            left.discard(name)
            if not left:
                needs_nth = True
                break
    step = node.name
    if is_target and node.name == "input" and node.get("type") and not node.get("id") and not node.get("class"):
        step += f"[type={escape(node['type'])}]"
    if needs_nth:
        step += f":nth-child({own_index + 1})"
    elif needs_classes:
        step += "".join("." + escape(name) for name in own)
    return step, False

listing.html (page 1) · dom/listing.html

<!doctype html>
<html lang="en"><head><meta charset="utf-8"><title>Products, page 1 · Nimbu Bazaar</title>
<link rel="stylesheet" href="/static/site.css"><script src="/static/catalogue.js" defer></script>
</head><body><img src="/static/logo.svg" alt="Nimbu Bazaar">
<table id="products" data-api="/api/v1/products?page=1&amp;per_page=24">
<tr><th>Product</th><th>Price</th></tr>
<tr><td><a href="/products/NB-0001/">Basmati rice 250 g</a></td><td>₹25.00</td></tr>
<tr><td><a href="/products/NB-0002/">Basmati rice 500 g</a></td><td>₹40.00</td></tr>
<tr><td><a href="/products/NB-0003/">Basmati rice 1 kg</a></td><td>₹70.00</td></tr>
<tr><td><a href="/products/NB-0004/">Basmati rice 2 kg</a></td><td>₹130.00</td></tr>
<tr><td><a href="/products/NB-0005/">Basmati rice 5 kg</a></td><td>₹310.00</td></tr>
<tr><td><a href="/products/NB-0006/">Basmati rice 10 kg</a></td><td>₹610.00</td></tr>
<tr><td><a href="/products/NB-0007/">Toor dal 250 g</a></td><td>₹34.25</td></tr>
<tr><td><a href="/products/NB-0008/">Toor dal 500 g</a></td><td>₹58.50</td></tr>
<tr><td><a href="/products/NB-0009/">Toor dal 1 kg</a></td><td>₹107.00</td></tr>
<tr><td><a href="/products/NB-0010/">Toor dal 2 kg</a></td><td>₹204.00</td></tr>
<tr><td><a href="/products/NB-0011/">Toor dal 5 kg</a></td><td>₹495.00</td></tr>
<tr><td><a href="/products/NB-0012/">Toor dal 10 kg</a></td><td>₹980.00</td></tr>
<tr><td><a href="/products/NB-0013/">Green tea 250 g</a></td><td>₹43.50</td></tr>
<tr><td><a href="/products/NB-0014/">Green tea 500 g</a></td><td>₹77.00</td></tr>
<tr><td><a href="/products/NB-0015/">Green tea 1 kg</a></td><td>₹144.00</td></tr>
<tr><td><a href="/products/NB-0016/">Green tea 2 kg</a></td><td>₹278.00</td></tr>
<tr><td><a href="/products/NB-0017/">Green tea 5 kg</a></td><td>₹680.00</td></tr>
<tr><td><a href="/products/NB-0018/">Green tea 10 kg</a></td><td>₹1,350.00</td></tr>
<tr><td><a href="/products/NB-0019/">Jaggery 250 g</a></td><td>₹52.75</td></tr>
<tr><td><a href="/products/NB-0020/">Jaggery 500 g</a></td><td>₹95.50</td></tr>
<tr><td><a href="/products/NB-0021/">Jaggery 1 kg</a></td><td>₹181.00</td></tr>
<tr><td><a href="/products/NB-0022/">Jaggery 2 kg</a></td><td>₹352.00</td></tr>
<tr><td><a href="/products/NB-0023/">Jaggery 5 kg</a></td><td>₹865.00</td></tr>
<tr><td><a href="/products/NB-0024/">Jaggery 10 kg</a></td><td>₹1,720.00</td></tr>
</table></body></html>

Both copied selectors match once in the browser’s tree and never in html.parser’s, because each passes through the inserted tbody. They are also fragile in a second way: tr:nth-child(2) names a position, so a new product at the top of the list makes the same selector point at another row. The selector written by hand anchors on the table’s id and on the shape of the product links, and finds all 24 links in both trees.

Key idea

Test every selector on the HTML your collector downloads: the Elements panel shows the browser’s repaired, script-changed tree, so a selector that works there can match nothing in the bytes.

Checkpoint

Check: DevTools views and copied selectors

Question 1 of 2 Following DevTools' Copy selector rules, what does it copy for the price cell of the second product, Basmati rice 500 g?
Choose one answer.

#products > tbody > tr:nth-child(3) > td:nth-child(2) #products > tr:nth-child(3) > td:nth-child(2) html > body > table > tbody > tr > td #products > tbody > tr:nth-child(2) > td:nth-child(2)

Show the answer: Question 1 of 2

Answer: #products > tbody > tr:nth-child(3) > td:nth-child(2)

cssPath adds one step per parent, joined with >, gives a position when a sibling has the same tag and no class tells them apart, and stops at an id: here a, then td:nth-child(2), tr:nth-child(3), tbody and #products.
Question 2 of 2 Which view shows the bytes that your collector downloads?
Choose one answer.

The search box of the Elements panel The Elements panel The Network panel filtered to Fetch/XHR View source, or the document's response in the Network panel

Show the answer: Question 2 of 2

Answer: View source, or the document's response in the Network panel

The page source and the document request's response hold the bytes as sent. Everything in the Elements panel, its search and its Copy menu included, works on the live DOM.

Practise: walk the tree yourself

The exercise builds the stack of open elements from this lesson’s first section. Write element_path(), which returns the tag names from the outermost element down to a piece of text; the sample tests include the start of the catalogue page, where <meta>, <link> and <img> must never stay open:

Exercise · Easy · Python

Write the path from the root to a piece of text

Write element_path(html, text) in path.py. It reads html with Python's own html.parser and returns the list of tag names from the outermost element down to the element that holds text, such as ["ul", "li", "a"], or None when no text node equals text.

A text node equals text when its characters, with spaces and line breaks at both ends removed, are the same.
Keep a stack of the elements that are open: handle_starttag pushes, handle_endtag pops back to the element it closes, and handle_data reads the stack when it sees the text.
Void elements (area, base, br, col, embed, hr, img, input, link, meta, source, track and wbr) never have an end tag, so they never go on the stack.
Return the path of the first matching text node.

The sample tests use a product list, a card with an image and line breaks, and the start of the practice shop's catalogue page, whose table has no <tbody> in its HTML.

Starter code · path.py

from html.parser import HTMLParser


class PathFinder(HTMLParser):
    def __init__(self, text):
        super().__init__()
        self.text = text
        self.open = []  # the names of the elements that are open, outermost first
        self.found = None

    def handle_starttag(self, tag, attrs):
        # Replace this: void elements such as <img> and <meta> are never closed, so they must not be pushed.
        self.open.append(tag)

    def handle_endtag(self, tag):
        if tag in self.open:
            while self.open.pop() != tag:
                pass

    def handle_data(self, data):
        if self.found is None and data.strip() == self.text:
            self.found = list(self.open)


def element_path(html, text):
    """The tag names from the outermost element to the element whose text node equals text, or None."""
    finder = PathFinder(text)
    finder.feed(html)
    finder.close()
    return finder.found

The sample tests · test_path.py

from path import element_path

SHOP_PAGE = """<!doctype html>
<html lang="en"><head><meta charset="utf-8"><title>Products, page 1 · Nimbu Bazaar</title>
<link rel="stylesheet" href="/static/site.css"><script src="/static/catalogue.js" defer></script>
</head><body><img src="/static/logo.svg" alt="Nimbu Bazaar">
<table id="products" data-api="/api/v1/products?page=1&amp;per_page=24">
<tr><th>Product</th><th>Price</th></tr>
<tr><td><a href="/products/NB-0001/">Basmati rice 250 g</a></td><td>₹25.00</td></tr>
<tr><td><a href="/products/NB-0002/">Basmati rice 500 g</a></td><td>₹40.00</td></tr>
</table></body></html>
"""


def test_a_simple_list():
    """the path names every open element, outermost first"""
    html = '<ul class="grid"><li><a href="/products/NB-0013/">Green tea 250 g</a></li></ul>'
    assert element_path(html, "Green tea 250 g") == ["ul", "li", "a"]


def test_void_elements_are_not_on_the_path():
    """an <img> or a <br> is never closed, so it never stays open"""
    html = '<div class="card"><img src="/img/tea.png" alt=""><p>Price<br>list</p><span>₹43.50</span></div>'
    assert element_path(html, "₹43.50") == ["div", "span"], "<img> and <br> have no end tag: they must not stay on the stack"
    assert element_path(html, "list") == ["div", "p"]


def test_the_shop_page_has_no_tbody():
    """html.parser's tree follows the markup as written: no <tbody>, which only a browser would add"""
    assert element_path(SHOP_PAGE, "Basmati rice 250 g") == ["html", "body", "table", "tr", "td", "a"], "<meta>, <link> and <img> are void elements"
    assert element_path(SHOP_PAGE, "₹40.00") == ["html", "body", "table", "tr", "td"]


def test_spaces_around_the_text_do_not_count():
    """a text node matches when it equals the text once spaces and line breaks at its ends are removed"""
    assert element_path("<p>\n  Jaggery 1 kg\n</p>", "Jaggery 1 kg") == ["p"]


def test_a_closed_element_leaves_the_path():
    """an end tag closes its own element, so a later sibling's path does not include it"""
    html = "<div><div><span>first</span></div><span>second</span></div>"
    assert element_path(html, "second") == ["div", "span"]
    assert element_path(html, "first") == ["div", "div", "span"]


def test_text_that_is_not_there():
    """no text node equals the text: None"""
    assert element_path(SHOP_PAGE, "Basmati rice 25 kg") is None

A hint

The starting code pushes every start tag. Find where <meta>, <link> and <img> end up on the stack of the shop's page: none of them is ever closed, so every path after them carries their names. Keep a set of the void elements and skip them in handle_starttag.

The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.

Then test yourself on trees, repairs, scripts and the DevTools views. Practice opens with whichever of the four your earlier answers got wrong most often:

Interview questions

Warm-up (fresher): What is the DOM, and why can it differ from the page source?

The DOM, the Document Object Model, is the tree of nodes that a browser or a parser builds from a page: element nodes with their attributes, text nodes and comment nodes, in document order. Programs read and change the page through it. The page source is the HTML the server sent, byte for byte. Two things make them differ. First, the browser builds the DOM by the HTML Standard’s parsing rules, which repair markup as they go: a table’s rows end up inside a tbody element even when the HTML has no <tbody> tag. Second, scripts change the DOM after parsing, adding or removing nodes, often with data fetched from an API. A scraper that downloads the HTML sees the source; a selector copied from the browser’s Elements panel describes the DOM, so it can fail on the source.

Two harder questions start from this lesson: a copied selector that finds nothing in a script, and a page whose products appear in the browser but not in the download. You will find where each answer begins, and what an interviewer would ask next, on the question page of this module:

Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.

Choose a pass

Fresher to mid

A selector you copied from Chrome's DevTools finds the price on a product page in your browser, but your Python script, parsing the downloaded page with Beautiful Soup and html.parser, gets None. How do you work out why, and what do you change?

Model answer, Pro. Model answers open with any pass; prices are in the notice on this page. First step and follow-ups
Fresher to mid

On a category page your browser shows 48 products, but the HTML your collector downloads contains only 12. Walk through how you find where the other 36 come from, and how you decide what the collector should fetch instead.

Model answer, Pro. First step and follow-ups

Key takeaways

A page is a tree of element, text and comment nodes, built in document order with a stack of open elements; attributes belong to their element, and whitespace between tags is text too.
Entities arrive decoded in text nodes, so   becomes a no-break space that a plain comparison does not match.
A browser builds the DOM by the HTML Standard’s rules and inserts what the markup leaves out, such as the tbody around a table’s rows; Python’s html.parser keeps the markup as written, and html5lib builds the browser’s tree.
Scripts change the DOM after parsing, so the page source is what a collector downloads and the Elements panel is what the browser shows now.
A selector copied with Copy selector describes the live DOM, with every inserted step and positions such as :nth-child(2); anchor selectors on ids and attributes, and test them on the downloaded HTML.

Terms in this lesson

DevTools: The developer tools built into a browser: the Elements panel shows the live DOM, the Network panel every request the page made, and the Console runs code against the page.
DOM: Document Object Model: the tree of nodes that a browser or a parser builds from a page, with elements, text and comments as nodes and attributes stored on their elements. Scripts read and change the page through it.
text node: A node of the DOM that holds a run of characters between two tags, such as a product name, or the spaces and line break between two elements. Entities such as & are already decoded in it.
void element: An HTML element that has only a start tag and never any content, such as img, meta, link, br and input. A parser closes it at once, so it never stays open around later elements.

References

HTML Standard: 13.2 Parsing HTML documents (13.2.6.4.9, the "in table" insertion mode) (WHATWG)
HTML Standard: 13.1 Writing HTML documents (void elements, optional tags) (WHATWG)
HTML Standard: 4.12.1 The script element (the defer attribute) (WHATWG)
DOM Standard: 4.2 Node tree (WHATWG)
html.parser: Simple HTML and XHTML parser (Python 3.14) (Python Software Foundation)
html5lib documentation (html5lib project)
Beautiful Soup documentation (version 4.15.0) (Beautiful Soup project)
Chrome DevTools: Get started with viewing and changing the DOM (Google (Chrome for Developers))
Chrome DevTools: Network features reference (Google (Chrome for Developers))
Chrome keyboard shortcuts (page source, developer tools, find) (Google (Chrome Help))
Chrome DevTools source: DOMPath.ts (cssPath, used by Copy selector) (Chromium DevTools project)

Report a problem with this lesson

In this module: Parsing HTML in Python and JavaScript

HTML, the DOM and DevTools for scrapers Free lesson, Pro answers: 20 minutes
BeautifulSoup 4.15: find, select, get_text Free core, Pro depth: 5 of 15 minutes free
html.parser, lxml or html5lib: why results differ Free core, Pro depth: 6 of 15 minutes free
CSS selectors that survive redesigns Free core, Pro depth: 7 of 20 minutes free
XPath 1.0 in Python: lxml and parsel Free core, Pro depth: 8 of 20 minutes free
Tables, lists and links: clean rows from HTML Free core, Pro depth: 7 of 20 minutes free
Parsing in JavaScript: cheerio and the DOM Free core, Pro depth: 7 of 20 minutes free

Your country

HTML, the DOM and DevTools for scrapers

What you will learn

Before you start

What you will learn

Before you start

A page is a tree of elements, text and comments

A browser repairs the markup before you see it

Scripts can change the DOM after the HTML arrives

DevTools answers four questions in three places

Practise: walk the tree yourself

Write the path from the root to a piece of text

Results of the sample tests

Interview questions

A selector you copied from Chrome's DevTools finds the price on a product page in your browser, but your Python script, parsing the downloaded page with Beautiful Soup and html.parser, gets None. How do you work out why, and what do you change?

On a category page your browser shows 48 products, but the HTML your collector downloads contains only 12. Walk through how you find where the other 36 come from, and how you decide what the collector should fetch instead.

Key takeaways

References

Related tools