Web Scraping & Browser Automation (Python and JavaScript) Module 3 – Parsing HTML in Python and JavaScript
CSS selectors that survive redesigns
Write CSS selectors with attributes, combinators, :has(), :nth-child() and :not(), rank the hooks they rely on, and alert when a selector stops matching.
What you will learn
- Write CSS selectors with attributes, combinators, :has(), :nth-child() and :not()
- Rank selector hooks by how long they last, from ids and data attributes down to classes and position
- Read each product's fields inside its own card, so that a missing field cannot shift the others
- Detect a selector that stops matching, and alert on the drop without alerting on normal changes
Before you start
What you’ll learn · Before you start
What you will learn
- Write CSS selectors with attributes, combinators, :has(), :nth-child() and :not()
- Rank selector hooks by how long they last, from ids and data attributes down to classes and position
- Read each product's fields inside its own card, so that a missing field cannot shift the others
- Detect a selector that stops matching, and alert on the drop without alerting on normal changes
Before you start
On this page
Last night the selector .price found 24 prices on the shop’s card view. Tonight the shop ships a redesign with new
class names, the same selector finds 0, and nothing raises an error: the collector simply stores a night without
prices.
A selector is a promise about a page’s markup, and some promises outlive a redesign far longer than others. Here you write selectors with every part a scraper needs, rank the hooks they can rely on, and test two of them against the redesigned page.
A selector names elements by what they are and where they sit
A CSS selector: A pattern that picks elements from a tree by tag, class, id, attribute, position or relation, such as #products a[href^="/products/"]. Browsers, Beautiful Soup and lxml all accept them. combines simple parts. The parts that relate two elements are
combinator: The part of a CSS selector that relates two elements: a space for any descendant, > for a direct child, + for the next sibling and ~ for any later sibling., and the parts that start with a colon are
pseudo-class: A part of a CSS selector that starts with a colon and tests a position, a state or a relation, such as :nth-child(2), :not(.out) or :has(.price).. Each part below matches something on the card view, where every
product is an li with a data-sku attribute, an itemprop="name", a price and a stock line:
| Part | Example | Matches |
|---|---|---|
| type, class, id | li, .price, #catalogue |
elements by tag name, class name or id |
| attribute | [data-sku], a[href^="/products/NB-001"] |
an attribute that exists, or whose value starts with /products/NB-001 |
| combinator | #catalogue li, #catalogue > li |
any descendant, with a space, or a direct child, with > |
| pseudo-class | li:nth-child(2), li:not(...), li:has(.stock.out) |
a position, a negation, or a card that contains a match |
:has() is the one that changes what scrapers can say: li:has(.stock.out) is a card that contains an
out-of-stock line, so li:has(.stock.out) [itemprop="name"] reads the names of the sold-out products. Follow how
that selector narrows 24 names down to 2:
The drawing shows the whole scene; its 4 steps are listed below.
Every step as text
- The rightmost part first. The element a selector returns is the one its last part describes: here an element with itemprop="name". Every card has one, so 24 elements are candidates.
- The space before it. The space is the descendant combinator: each name must sit somewhere inside an element matched by the part on its left. All 24 names are inside an li.
- The card must contain a line. :has(.stock.out) keeps only the li elements that contain an element with both classes, stock and out. Two cards do: the products sold out this week.
- Two names remain. Only the names inside those two cards match the whole selector. Remove :has(...) and you get all 24 again; put :not() around it and you get the names of the 22 cards in stock.
The program runs ten selectors on the card view with Beautiful Soup, whose select() hands them to Soup Sieve:
selectors/selector_basics.py
"""Ten selectors on page 1 of the catalogue's card view, run by Beautiful Soup 4.15.0 (Soup Sieve 2.10)."""
from pathlib import Path
from bs4 import BeautifulSoup
soup = BeautifulSoup(Path("grid_a.html").read_text(encoding="utf-8"), "html.parser")
SELECTORS = [
"li",
".price",
"#catalogue",
"[data-sku]",
'a[href^="/products/NB-001"]',
"#catalogue li",
"#catalogue > li",
'li:nth-child(2) [itemprop="name"]',
"li:not(:has(.stock.out))",
'li:has(.stock.out) [itemprop="name"]',
]
print(f"{'selector':<38} matches")
for selector in SELECTORS:
print(f"{selector:<38} {len(soup.select(selector)):>7}")
print("names of the sold-out products:", [t.get_text() for t in soup.select(SELECTORS[-1])])
print("second card's name:", soup.select_one(SELECTORS[7]).get_text())
print("product links NB-0010 to NB-0019:", [a["href"] for a in soup.select(SELECTORS[4])][:2], "...") Output
selector matches li 24 .price 24 #catalogue 1 [data-sku] 24 a[href^="/products/NB-001"] 10 #catalogue li 24 #catalogue > li 0 li:nth-child(2) [itemprop="name"] 1 li:not(:has(.stock.out)) 22 li:has(.stock.out) [itemprop="name"] 2 names of the sold-out products: ['Toor dal 5 kg', 'Jaggery 5 kg'] second card's name: Basmati rice 500 g product links NB-0010 to NB-0019: ['/products/NB-0010/', '/products/NB-0011/'] ...
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 selector_basics.py
grid_a.html (version A) · selectors/grid_a.html
<!doctype html>
<html lang="en">
<head><meta charset="utf-8"><title>Products, page 1 · Nimbu Bazaar</title></head>
<body>
<main id="catalogue">
<h1>Products</h1>
<ul class="grid">
<li class="product" data-sku="NB-0001" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0001/" itemprop="url"><span itemprop="name">Basmati rice 250 g</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="25.00">₹25.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0002" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0002/" itemprop="url"><span itemprop="name">Basmati rice 500 g</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="40.00">₹40.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0003" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0003/" itemprop="url"><span itemprop="name">Basmati rice 1 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="70.00">₹70.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0004" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0004/" itemprop="url"><span itemprop="name">Basmati rice 2 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="130.00">₹130.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0005" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0005/" itemprop="url"><span itemprop="name">Basmati rice 5 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="310.00">₹310.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0006" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0006/" itemprop="url"><span itemprop="name">Basmati rice 10 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="610.00">₹610.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0007" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0007/" itemprop="url"><span itemprop="name">Toor dal 250 g</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="34.25">₹34.25</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0008" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0008/" itemprop="url"><span itemprop="name">Toor dal 500 g</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="58.50">₹58.50</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0009" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0009/" itemprop="url"><span itemprop="name">Toor dal 1 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="107.00">₹107.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0010" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0010/" itemprop="url"><span itemprop="name">Toor dal 2 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="204.00">₹204.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0011" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0011/" itemprop="url"><span itemprop="name">Toor dal 5 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="495.00">₹495.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock out">Out of stock</span>
</li>
<li class="product" data-sku="NB-0012" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0012/" itemprop="url"><span itemprop="name">Toor dal 10 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="980.00">₹980.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0013" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0013/" itemprop="url"><span itemprop="name">Green tea 250 g</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="43.50">₹43.50</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0014" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0014/" itemprop="url"><span itemprop="name">Green tea 500 g</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="77.00">₹77.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0015" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0015/" itemprop="url"><span itemprop="name">Green tea 1 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="144.00">₹144.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0016" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0016/" itemprop="url"><span itemprop="name">Green tea 2 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="278.00">₹278.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0017" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0017/" itemprop="url"><span itemprop="name">Green tea 5 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="680.00">₹680.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0018" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0018/" itemprop="url"><span itemprop="name">Green tea 10 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="1350.00">₹1,350.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0019" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0019/" itemprop="url"><span itemprop="name">Jaggery 250 g</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="52.75">₹52.75</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0020" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0020/" itemprop="url"><span itemprop="name">Jaggery 500 g</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="95.50">₹95.50</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0021" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0021/" itemprop="url"><span itemprop="name">Jaggery 1 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="181.00">₹181.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0022" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0022/" itemprop="url"><span itemprop="name">Jaggery 2 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="352.00">₹352.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0023" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0023/" itemprop="url"><span itemprop="name">Jaggery 5 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="865.00">₹865.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock out">Out of stock</span>
</li>
<li class="product" data-sku="NB-0024" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0024/" itemprop="url"><span itemprop="name">Jaggery 10 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="1720.00">₹1,720.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
</ul>
</main>
</body>
</html> #catalogue li found the 24 cards and #catalogue > li none, because the cards are children of the ul, not of
main. The attribute prefix picked the ten products from NB-0010 to NB-0019, and :not(:has(...)) kept the 22
cards that are in stock. lxml.html and Scrapy run CSS through another engine, cssselect, which translates each
selector into XPath.
Checkpoint
Check: The parts of a CSS selector
Hooks differ in how long they last
Every selector leans on a hook in the markup, and a hook lasts as long as the site has a reason to keep it. From the longest-lived to the shortest:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- An id. An id names one element on purpose, and the page's own links, scripts and styles may point at it, so it is the hook a page has most reason to keep.
- A data attribute. Attributes such as data-sku or a test id exist for the site's own code and tests, which break if they change, so they tend to outlive a redesign.
- Microdata and semantics. itemprop="price", ARIA roles and labels serve search engines and assistive technology. A redesign that keeps those working keeps these hooks too.
- Text. Text such as a rupee sign changes rarely, but it matches every element that contains it, the wrappers around a price included, so it over-matches.
- A styling class. A class exists for the style sheet. A redesign renames classes freely, often into generated names, and every selector built on the old names breaks with them.
- A position. A position such as nth-child(2), or a chain of > steps copied from DevTools, changes with every inserted card, banner or wrapper.
- An id names one element on purpose, so a page rarely renames it.
- A data attribute, such as
data-skuor a test id, exists for the site’s own scripts or tests, which break if it changes. - Microdata and other semantics, such as
itemprop="price", roles and labels, are kept for search engines and assistive technology. - Text, such as the rupee sign, changes rarely, but matches every element that contains it.
- A class is there for the style sheet, and a redesign renames classes freely, often into generated names.
- A position, such as
:nth-child(2), changes with every inserted card or wrapper.
The program tests two hooks that price selectors often use, a class and microdata, on version B, the redesign:
versions/two_versions.py
"""Two selectors for the price, written for the card view (version A), run on its redesign (version B) too."""
from pathlib import Path
from bs4 import BeautifulSoup
pages = {v: BeautifulSoup(Path(f"grid_{v.lower()}.html").read_text(encoding="utf-8"), "html.parser") for v in "AB"}
for selector in (".price", '[itemprop="price"]'):
counts = ", ".join(f"version {v}: {len(soup.select(selector))}" for v, soup in pages.items())
print(f"{selector:<20} {counts}")
print("products on version B:", len(pages["B"].select("[data-sku]")),
"| sold out, no price shown:", len(pages["B"].select("[data-sku]:not(:has([itemprop='price']))"))) Output
.price version A: 24, version B: 0 [itemprop="price"] version A: 24, version B: 22 products on version B: 24 | sold out, no price shown: 2
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 two_versions.py
grid_a.html (version A) · versions/grid_a.html
<!doctype html>
<html lang="en">
<head><meta charset="utf-8"><title>Products, page 1 · Nimbu Bazaar</title></head>
<body>
<main id="catalogue">
<h1>Products</h1>
<ul class="grid">
<li class="product" data-sku="NB-0001" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0001/" itemprop="url"><span itemprop="name">Basmati rice 250 g</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="25.00">₹25.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0002" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0002/" itemprop="url"><span itemprop="name">Basmati rice 500 g</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="40.00">₹40.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0003" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0003/" itemprop="url"><span itemprop="name">Basmati rice 1 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="70.00">₹70.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0004" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0004/" itemprop="url"><span itemprop="name">Basmati rice 2 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="130.00">₹130.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0005" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0005/" itemprop="url"><span itemprop="name">Basmati rice 5 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="310.00">₹310.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0006" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0006/" itemprop="url"><span itemprop="name">Basmati rice 10 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="610.00">₹610.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0007" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0007/" itemprop="url"><span itemprop="name">Toor dal 250 g</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="34.25">₹34.25</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0008" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0008/" itemprop="url"><span itemprop="name">Toor dal 500 g</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="58.50">₹58.50</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0009" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0009/" itemprop="url"><span itemprop="name">Toor dal 1 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="107.00">₹107.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0010" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0010/" itemprop="url"><span itemprop="name">Toor dal 2 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="204.00">₹204.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0011" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0011/" itemprop="url"><span itemprop="name">Toor dal 5 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="495.00">₹495.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock out">Out of stock</span>
</li>
<li class="product" data-sku="NB-0012" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0012/" itemprop="url"><span itemprop="name">Toor dal 10 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="980.00">₹980.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0013" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0013/" itemprop="url"><span itemprop="name">Green tea 250 g</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="43.50">₹43.50</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0014" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0014/" itemprop="url"><span itemprop="name">Green tea 500 g</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="77.00">₹77.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0015" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0015/" itemprop="url"><span itemprop="name">Green tea 1 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="144.00">₹144.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0016" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0016/" itemprop="url"><span itemprop="name">Green tea 2 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="278.00">₹278.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0017" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0017/" itemprop="url"><span itemprop="name">Green tea 5 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="680.00">₹680.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0018" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0018/" itemprop="url"><span itemprop="name">Green tea 10 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="1350.00">₹1,350.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0019" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0019/" itemprop="url"><span itemprop="name">Jaggery 250 g</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="52.75">₹52.75</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0020" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0020/" itemprop="url"><span itemprop="name">Jaggery 500 g</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="95.50">₹95.50</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0021" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0021/" itemprop="url"><span itemprop="name">Jaggery 1 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="181.00">₹181.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0022" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0022/" itemprop="url"><span itemprop="name">Jaggery 2 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="352.00">₹352.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0023" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0023/" itemprop="url"><span itemprop="name">Jaggery 5 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="865.00">₹865.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock out">Out of stock</span>
</li>
<li class="product" data-sku="NB-0024" itemscope itemtype="https://schema.org/Product">
<a class="name" href="/products/NB-0024/" itemprop="url"><span itemprop="name">Jaggery 10 kg</span></a>
<span class="price" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="1720.00">₹1,720.00</span><meta itemprop="priceCurrency" content="INR"></span>
<span class="stock in">In stock</span>
</li>
</ul>
</main>
</body>
</html> grid_b.html (version B) · versions/grid_b.html
<!doctype html>
<html lang="en">
<head><meta charset="utf-8"><title>Products, page 1 · Nimbu Bazaar</title></head>
<body>
<div id="__app">
<header class="Header_bar__Te0q"><h1>Products</h1></header>
<section class="ProductGrid_grid__k2Lx" aria-label="Products">
<article class="Card_card__3hQ1" data-sku="NB-0001" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0001.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0001/" itemprop="url"><span itemprop="name">Basmati rice 250 g</span></a></h3>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="25.00">₹25.00</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0002" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0002.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0002/" itemprop="url"><span itemprop="name">Basmati rice 500 g</span></a></h3>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="40.00">₹40.00</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0003" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0003.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0003/" itemprop="url"><span itemprop="name">Basmati rice 1 kg</span></a></h3>
<div class="Card_badges__x1Rf"><span class="Badge_badge__aa1Z">New</span></div>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="70.00">₹70.00</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0004" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0004.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0004/" itemprop="url"><span itemprop="name">Basmati rice 2 kg</span></a></h3>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="130.00">₹130.00</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0005" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0005.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0005/" itemprop="url"><span itemprop="name">Basmati rice 5 kg</span></a></h3>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="310.00">₹310.00</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0006" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0006.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0006/" itemprop="url"><span itemprop="name">Basmati rice 10 kg</span></a></h3>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="610.00">₹610.00</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0007" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0007.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0007/" itemprop="url"><span itemprop="name">Toor dal 250 g</span></a></h3>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="34.25">₹34.25</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0008" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0008.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0008/" itemprop="url"><span itemprop="name">Toor dal 500 g</span></a></h3>
<div class="Card_badges__x1Rf"><span class="Badge_badge__aa1Z">New</span></div>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="58.50">₹58.50</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0009" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0009.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0009/" itemprop="url"><span itemprop="name">Toor dal 1 kg</span></a></h3>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="107.00">₹107.00</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0010" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0010.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0010/" itemprop="url"><span itemprop="name">Toor dal 2 kg</span></a></h3>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="204.00">₹204.00</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0011" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0011.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0011/" itemprop="url"><span itemprop="name">Toor dal 5 kg</span></a></h3>
<p class="Card_stock__L0pY">Sold out</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0012" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0012.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0012/" itemprop="url"><span itemprop="name">Toor dal 10 kg</span></a></h3>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="980.00">₹980.00</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0013" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0013.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0013/" itemprop="url"><span itemprop="name">Green tea 250 g</span></a></h3>
<div class="Card_badges__x1Rf"><span class="Badge_badge__aa1Z">New</span></div>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="43.50">₹43.50</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0014" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0014.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0014/" itemprop="url"><span itemprop="name">Green tea 500 g</span></a></h3>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="77.00">₹77.00</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0015" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0015.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0015/" itemprop="url"><span itemprop="name">Green tea 1 kg</span></a></h3>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="144.00">₹144.00</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0016" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0016.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0016/" itemprop="url"><span itemprop="name">Green tea 2 kg</span></a></h3>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="278.00">₹278.00</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0017" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0017.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0017/" itemprop="url"><span itemprop="name">Green tea 5 kg</span></a></h3>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="680.00">₹680.00</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0018" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0018.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0018/" itemprop="url"><span itemprop="name">Green tea 10 kg</span></a></h3>
<div class="Card_badges__x1Rf"><span class="Badge_badge__aa1Z">New</span></div>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="1350.00">₹1,350.00</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0019" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0019.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0019/" itemprop="url"><span itemprop="name">Jaggery 250 g</span></a></h3>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="52.75">₹52.75</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0020" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0020.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0020/" itemprop="url"><span itemprop="name">Jaggery 500 g</span></a></h3>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="95.50">₹95.50</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0021" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0021.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0021/" itemprop="url"><span itemprop="name">Jaggery 1 kg</span></a></h3>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="181.00">₹181.00</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0022" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0022.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0022/" itemprop="url"><span itemprop="name">Jaggery 2 kg</span></a></h3>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="352.00">₹352.00</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0023" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0023.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0023/" itemprop="url"><span itemprop="name">Jaggery 5 kg</span></a></h3>
<div class="Card_badges__x1Rf"><span class="Badge_badge__aa1Z">New</span></div>
<p class="Card_stock__L0pY">Sold out</p>
</article>
<article class="Card_card__3hQ1" data-sku="NB-0024" itemscope itemtype="https://schema.org/Product">
<div class="Card_media__9sPz"><img src="/img/NB-0024.webp" alt=""></div>
<h3 class="Card_title__Qm7R"><a href="/products/NB-0024/" itemprop="url"><span itemprop="name">Jaggery 10 kg</span></a></h3>
<div class="Card_price__Zr8K" data-testid="price" itemprop="offers" itemscope itemtype="https://schema.org/Offer"><span itemprop="price" content="1720.00">₹1,720.00</span><meta itemprop="priceCurrency" content="INR"></div>
<p class="Card_stock__L0pY">In stock</p>
</article>
</section>
</div>
</body>
</html> The class found no price on version B, where every class name changed. The microdata hook found 22, and 22 is right: the two sold-out products show no price at all on the redesigned page.
Key idea
Anchor a selector on the hook the site has a reason to keep, such as an id, a data attribute or microdata, and fall back to styling classes and positions only when the page offers nothing better.
Checkpoint
Check: Hooks that last, and hooks that do not
Six hooks for the price, measured on a redesign Pro: opens with a Pro, Premium or Ultimate pass
Each rung of the ladder can be measured. This part runs six price selectors, one per kind of hook, on both versions of the page, explains each failure, and checks which selectors work in the other selector engine Python scrapers meet.
In this part:
- Six selectors, two versions, one table
- Selectors that work in one engine only
2 runnable examples (grid_a.html (version A), grid_b.html (version B), portability.py, selector_ladder.py) · about 4 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Scope each product, then read its fields Pro: opens with a Pro, Premium or Ultimate pass
A good selector can still give a wrong record when fields are read across the whole page. This part reproduces how one missing price shifts every price after it, and reads the same page one card at a time.
In this part:
- Two missing prices, fourteen wrong records
- The rule: find the products, then read inside each
1 runnable example (grid_b.html (version B), scoped_fields.py) · about 4 minutes
Opening this part with your pass…Pro part. It opens with the same pass as the part above.
Match counts that raise an alarm before bad data is stored Pro: opens with a Pro, Premium or Ultimate pass
A redesign shows up first as a change in how many elements each selector matches. This part records those counts on every run, alerts when one falls, and builds the alert function as an exercise.
In this part:
- Counting matches on every run
- Practise: write the alert
- Raise an alert when a selector's matches fall
- Results of the sample tests
1 runnable example (grid_a.html (version A), grid_b.html (version B), match_counts.py) · 1 exercise with sample tests · about 4 minutes
Opening this part with your pass…Pro part. It opens with the same pass as the part above.
Practise writing and ranking selectors
Test yourself on selector parts and on the hooks they rely on. Your weakest idea from the checkpoints comes up first:
Practice that adapts to you
Practise this lesson
7 questions about 5 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
Warm-up (fresher): What is the difference between the selectors div p and div > p?
Both select p elements, but they ask for a different relation. div p, with a space, the descendant combinator,
matches a paragraph at any depth inside a div, inside a section or a list item as well. div > p, the child
combinator, matches only a paragraph whose parent is the div itself. A scraper usually wants the space, because
wrappers come and go; > is useful when a nested element of the same kind must be left out.
Three harder questions start from this lesson: a scraper that broke after a redesign, a choice between a class and a data attribute, and a monitor for selectors on many sites. Where to begin each answer, and the follow-ups worth preparing, are on the question page of this module:
-
Mid to senior
Overnight a shop redesigned its listing pages and your collector stored a night of products without prices, without any error. How do you repair the collector, and what do you change so the next redesign is caught before bad data is stored?
-
Mid to senior
A teammate selects prices with .ProductCard_price__x7Q2a, the class DevTools shows; you propose [data-sku] [itemprop="price"]. Make the case for one of them, with the evidence you would collect.
-
Mid to senior
You run collectors for 50 sites, each with about ten selectors. Design the checks that tell you, on the night it happens, that a site changed its markup, without alerting so often that people stop reading the alerts.
Key takeaways
- A selector combines type, class, id, attribute and pseudo-class parts with combinators;
:has()selects an element by what it contains. - Hooks last as long as the site needs them: ids, data attributes and microdata outlast styling classes and positions.
- Test a selector on the page before and after a change: the class
.pricefound 0 of 24 prices after the redesign, and the microdata hook found all 22 that exist.
Terms in this lesson
- combinator
- The part of a CSS selector that relates two elements: a space for any descendant, > for a direct child, + for the next sibling and ~ for any later sibling.
- CSS selector
- A pattern that picks elements from a tree by tag, class, id, attribute, position or relation, such as #products a[href^="/products/"]. Browsers, Beautiful Soup and lxml all accept them.
- pseudo-class
- A part of a CSS selector that starts with a colon and tests a position, a state or a relation, such as :nth-child(2), :not(.out) or :has(.price).
References
- Soup Sieve documentation: selectors and pseudo-classes (Soup Sieve project)
- cssselect documentation (CSS selectors translated to XPath 1.0) (Scrapy project (cssselect))
- parsel API reference: Selector.css (CSS to XPath with cssselect) (Scrapy project (parsel))
- Scrapy: Selectors (a thin wrapper around parsel) (Scrapy project)
- Beautiful Soup documentation (version 4.15.0) (Beautiful Soup project)
- CSS selectors: attribute selectors (Mozilla (MDN Web Docs))
- :has() pseudo-class (Mozilla (MDN Web Docs))
- :nth-child() pseudo-class (Mozilla (MDN Web Docs))
- HTML Standard: microdata (itemscope, itemprop) (WHATWG)
- Built-in functions: zip() and its strict argument (Python 3.14) (Python Software Foundation)
- statistics: Mathematical statistics functions (median), Python 3.14 (Python Software Foundation)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress