Web Scraping & Browser Automation (Python and JavaScript) Module 3 – Parsing HTML in Python and JavaScript

BeautifulSoup 4.15: find, select, get_text

Parse HTML with Beautiful Soup 4.15: find and find_all, CSS selectors with select, clean text with get_text, attributes, and absolute links with urljoin.

Beginner
15 minutes
Examples run with Python 3.14.8, Pyodide 314.0.7, Node.js 24.21.0, quickjs 0.32.0, beautifulsoup4 4.15.0, soupsieve 2.10 and lxml 6.1.3
By MySmartCoPilot

Free: Beautiful Soup turns the HTML into a tree you can search · Search the tree by tag or by CSS selector · Text, attributes and links become fields. With a pass: 3 in-depth parts and 3 interview answers, about 10 minutes.

What you will learn

Parse a page with BeautifulSoup and a named parser, and find elements with find_all and CSS selectors
Extract text and attributes as clean fields, and resolve relative links with urljoin
Handle cards with missing or extra markup without a crash or a shifted value
Update old tutorial code that uses the deprecated camelCase methods and the text argument

Before you start

HTML, the DOM and DevTools for scrapers

What you’ll learn · Before you start

What you will learn

Parse a page with BeautifulSoup and a named parser, and find elements with find_all and CSS selectors
Extract text and attributes as clean fields, and resolve relative links with urljoin
Handle cards with missing or extra markup without a crash or a shifted value
Update old tutorial code that uses the deprecated camelCase methods and the text argument

Before you start

HTML, the DOM and DevTools for scrapers

On this page

Page 1 of the practice shop’s catalogue, in its card view, is 4,822 bytes of HTML with 24 products in it. About fifteen lines of Beautiful Soup turn it into 24 dictionaries, each with a name, a price, a stock flag and a link that works outside the page.

Beautiful Soup: A Python library that builds a searchable tree from HTML or XML with a parser you choose, and finds elements in it by tag, attributes, text or CSS selector. 4.15.0 builds the tree you met in the last lesson and gives it two ways to search, find_all() and select(), and one way to read, get_text(). Here you use all three on the card view, then turn each card into a record with an absolute link.

Beautiful Soup turns the HTML into a tree you can search

BeautifulSoup(html, "html.parser") parses the text with the parser you name and returns the document’s tree. Name it every time. Without a name, Beautiful Soup picks the best parser installed on that computer, and its documentation advises naming one when a script is shared or runs on several machines, because parsers build different trees from the same broken markup. "html.parser" is Python’s own and needs nothing installed.

Each element becomes a Tag with a name and a dictionary of attributes, and each run of text becomes a NavigableString, a kind of Python string. Step through the first card:

Interactive · step by step The first card as Beautiful Soup holds it Step from the document down to the first card's price, and see which object each part of the tree becomes.

The drawing shows the whole scene; its 6 steps are listed below.

Every step as text

The document. BeautifulSoup(html, "html.parser") returns one object for the whole document, at the top of the tree. Searches called on it look through every element of the page.
A Tag for each element. Each element becomes a Tag. The first card is a Tag named li whose attributes form a dictionary: card["data-sku"] is "NB-0001", and card["class"] is the list ["product"], because a class attribute can hold several names.
Children are Tags too. The link inside the card is another Tag, a, with its own class and href. card.find("a") starts its search at the card, so it finds this link and never another card's.
Text is a NavigableString. The link's text is a NavigableString, a subclass of Python's str, so every string method works on it. Here it is the name, Basmati rice 250 g.
The price, one step down. The price is a span Tag with the class price, and its text, ₹25.00, is one more NavigableString. card.select_one(".price") reaches it with a CSS selector, again only inside this card.
One card, one record. Read the link's text, its href and the price's text from the same card, and the three values belong to the same product. The loop below does that for all 24 cards.

soup.title is a shortcut for the first <title> in the tree, and a search called on a tag looks only below it, which is what keeps one card’s fields together.

Search the tree by tag or by CSS selector

Two families of methods search the tree, and on this page they find the same 24 cards:

Method	Matches by	Returns
`find_all("li", class_="product")`	tag name, attributes, text or a function	a list, empty when nothing matches
`find("a", class_="name")`	the same	the first match, or `None`
`select("li.product")`	a CSS selector	a list, empty when nothing matches
`select_one(".price")`	a CSS selector	the first match, or `None`

find_all() takes Python arguments: a tag name, class_ (with an underscore, because class is a Python keyword), any other attribute as a keyword, or an attrs dictionary. select() takes the CSS selectors you test in a browser’s DevTools, run by Soup Sieve, the selector library that Beautiful Soup uses for these two methods. Both search below the tag you call them on, so card.find("a") looks inside one card only.

Checkpoint

Check: Parsing with a named parser · find_all, find, select and select_one

Question 1 of 2 A script calls BeautifulSoup(html) without naming a parser. It works on your laptop, which has lxml installed. What changes when it runs on a server without lxml?
Choose one answer.

Beautiful Soup picks another installed parser, which can build a different tree from broken markup It stops with an error that lxml is missing It downloads lxml the first time it runs Nothing, because Beautiful Soup has its own built-in parser

Show the answer: Question 1 of 2

Answer: Beautiful Soup picks another installed parser, which can build a different tree from broken markup

BeautifulSoup(html) without a parser name means "the best parser installed here", and that differs between computers. BeautifulSoup(html, "html.parser"), or the parser you chose, gives the same tree everywhere.
Question 2 of 2 Which call finds every product card of the card view, the li elements with the class product?
Choose one answer.

soup.find_all("li", class_="product") soup.find_all("li", class="product") soup.select("li product") soup.find("li", class_="product")

Show the answer: Question 2 of 2

Answer: soup.find_all("li", class_="product")

Two equivalent ways find the 24 cards: find_all("li", class_="product") with Python arguments, and select("li.product") with a CSS selector. The program printed 24 for both.

Text, attributes and links become fields

A field is the text of an element, or the value of one of its attributes. get_text(" ", strip=True) joins every text node below the element with one space and removes whitespace at the ends of each, so a name split across nested tags still reads as words. Beautiful Soup has three more ways to read text. .string is the element’s text only when it has exactly one child, and None otherwise. .text is get_text() with no arguments, and stripped_strings gives each piece of text on its own. tag["href"] reads an attribute and stops with a KeyError when it is missing, while tag.get("href") returns None.

Links in a page are often relative, such as /products/NB-0001/ or ?page=2. Store them as an absolute URL: An address with its scheme and host, such as https://bazaar.nimbu.test/products/NB-0001/. A relative link such as /products/NB-0001/ only has a meaning together with the page it came from.: urljoin(page_url, href) resolves them against the address the page came from. The program reads the card view, saved from the address in PAGE_URL:

24 cards into 24 records

grid/products_bs4.py

"""Page 1 of the catalogue's card view, turned into one dictionary per product with Beautiful Soup 4.15.0."""
import json
from pathlib import Path
from urllib.parse import urljoin

from bs4 import BeautifulSoup

PAGE_URL = "https://bazaar.nimbu.test/products/grid/?page=1"  # the address grid.html was saved from

soup = BeautifulSoup(Path("grid.html").read_text(encoding="utf-8"), "html.parser")
print("title:", soup.title.get_text())

cards = soup.find_all("li", class_="product")
print("find_all('li', class_='product'):", len(cards), "| select('li.product'):", len(soup.select("li.product")))

products = []
for card in cards:
    link = card.find("a", class_="name")
    products.append({
        "sku": card["data-sku"],
        "name": link.get_text(" ", strip=True),
        "price": card.select_one(".price").get_text(" ", strip=True),
        "in_stock": card.select_one(".stock.in") is not None,
        "url": urljoin(PAGE_URL, link["href"]),
    })

for p in products[:3]:
    print(json.dumps(p, ensure_ascii=False))
print(f"... {len(products)} products")

next_link = soup.find("a", rel="next")
print("href of the next-page link:", next_link["href"], "->", urljoin(PAGE_URL, next_link["href"]))

Output

title: Products, page 1 · Nimbu Bazaar
find_all('li', class_='product'): 24 | select('li.product'): 24
{"sku": "NB-0001", "name": "Basmati rice 250 g", "price": "₹25.00", "in_stock": true, "url": "https://bazaar.nimbu.test/products/NB-0001/"}
{"sku": "NB-0002", "name": "Basmati rice 500 g", "price": "₹40.00", "in_stock": true, "url": "https://bazaar.nimbu.test/products/NB-0002/"}
{"sku": "NB-0003", "name": "Basmati rice 1 kg", "price": "₹70.00", "in_stock": true, "url": "https://bazaar.nimbu.test/products/NB-0003/"}
... 24 products
href of the next-page link: ?page=2 -> https://bazaar.nimbu.test/products/grid/?page=2

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 products_bs4.py

grid.html (24 cards) · grid/grid.html

<!doctype html>
<html lang="en">
<head><meta charset="utf-8"><title>Products, page 1 · Nimbu Bazaar</title></head>
<body>
<main id="catalogue">
<h1>Products</h1>
<ul class="grid">
<li class="product" data-sku="NB-0001">
  <a class="name" href="/products/NB-0001/">Basmati rice 250 g</a>
  <span class="price">₹25.00</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0002">
  <a class="name" href="/products/NB-0002/">Basmati rice 500 g</a>
  <span class="price">₹40.00</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0003">
  <a class="name" href="/products/NB-0003/">Basmati rice 1 kg</a>
  <span class="price">₹70.00</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0004">
  <a class="name" href="/products/NB-0004/">Basmati rice 2 kg</a>
  <span class="price">₹130.00</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0005">
  <a class="name" href="/products/NB-0005/">Basmati rice 5 kg</a>
  <span class="price">₹310.00</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0006">
  <a class="name" href="/products/NB-0006/">Basmati rice 10 kg</a>
  <span class="price">₹610.00</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0007">
  <a class="name" href="/products/NB-0007/">Toor dal 250 g</a>
  <span class="price">₹34.25</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0008">
  <a class="name" href="/products/NB-0008/">Toor dal 500 g</a>
  <span class="price">₹58.50</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0009">
  <a class="name" href="/products/NB-0009/">Toor dal 1 kg</a>
  <span class="price">₹107.00</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0010">
  <a class="name" href="/products/NB-0010/">Toor dal 2 kg</a>
  <span class="price">₹204.00</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0011">
  <a class="name" href="/products/NB-0011/">Toor dal 5 kg</a>
  <span class="price">₹495.00</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0012">
  <a class="name" href="/products/NB-0012/">Toor dal 10 kg</a>
  <span class="price">₹980.00</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0013">
  <a class="name" href="/products/NB-0013/">Green tea 250 g</a>
  <span class="price">₹43.50</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0014">
  <a class="name" href="/products/NB-0014/">Green tea 500 g</a>
  <span class="price">₹77.00</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0015">
  <a class="name" href="/products/NB-0015/">Green tea 1 kg</a>
  <span class="price">₹144.00</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0016">
  <a class="name" href="/products/NB-0016/">Green tea 2 kg</a>
  <span class="price">₹278.00</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0017">
  <a class="name" href="/products/NB-0017/">Green tea 5 kg</a>
  <span class="price">₹680.00</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0018">
  <a class="name" href="/products/NB-0018/">Green tea 10 kg</a>
  <span class="price">₹1,350.00</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0019">
  <a class="name" href="/products/NB-0019/">Jaggery 250 g</a>
  <span class="price">₹52.75</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0020">
  <a class="name" href="/products/NB-0020/">Jaggery 500 g</a>
  <span class="price">₹95.50</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0021">
  <a class="name" href="/products/NB-0021/">Jaggery 1 kg</a>
  <span class="price">₹181.00</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0022">
  <a class="name" href="/products/NB-0022/">Jaggery 2 kg</a>
  <span class="price">₹352.00</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0023">
  <a class="name" href="/products/NB-0023/">Jaggery 5 kg</a>
  <span class="price">₹865.00</span>
  <span class="stock in">In stock</span>
</li>
<li class="product" data-sku="NB-0024">
  <a class="name" href="/products/NB-0024/">Jaggery 10 kg</a>
  <span class="price">₹1,720.00</span>
  <span class="stock in">In stock</span>
</li>
</ul>
<nav class="pages"><a href="?page=2" rel="next">Next page</a></nav>
</main>
</body>
</html>

find_all() and select() agreed on 24 cards, every price came out as written on the page, and ?page=2, a link that only names a query, became the full address of the next page.

Checkpoint

Check: Text, attributes and absolute links

Question 1 of 2 card.find("span", class_="badge").get_text() raises AttributeError: 'NoneType' object has no attribute 'get_text'. What happened?
Choose one answer.

This card has no matching span, so find returned None, and None has no get_text get_text only works on the whole soup, not on one card The span exists but holds no text The class badge must be written as class_="badge" inside get_text

Show the answer: Question 1 of 2

Answer: This card has no matching span, so find returned None, and None has no get_text

A single-result search gives None for a missing element. Test it (tag is None) before reading text or attributes, or use a list search, which is simply empty.
Question 2 of 2 The card view was downloaded from https://bazaar.nimbu.test/products/grid/?page=1 and its next-page link is href="?page=2". What does urljoin(page_url, href) return?
Choose one answer.

https://bazaar.nimbu.test/products/grid/?page=1?page=2 https://bazaar.nimbu.test/?page=2 https://bazaar.nimbu.test/products/grid/?page=2 ?page=2

Show the answer: Question 2 of 2

Answer: https://bazaar.nimbu.test/products/grid/?page=2

Store links absolute, resolved against the address the page came from. Relative links such as /products/NB-0001/ or ?page=2 mean nothing once the record is away from its page.

Clean text from messy markup: five traps, measured Pro: opens with a Pro, Premium or Ultimate pass

A shop’s own offer pages grow sale prices, units, line breaks and badges inside the elements you read. This part measures what each way of reading text returns on five such cards, and turns a price element’s text into a number you can store.

In this part:

Five ways to read text, on five awkward cards
Turning price text into a number you can store
Turn a price element's text into whole paise
- Results of the sample tests

1 runnable example (offers.html (10 cards), text_traps.py) · 1 exercise with sample tests · about 5 minutes

Opening this part with your pass…

This part opens with any MySmartCoPilot pass.

Pro pass · 1 week₹79$2.99
30 days₹179$5.99
1 year₹1,699$45

One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.

Choose a pass I have a licence key

It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).

Pages where fields go missing, read without crashes Pro: opens with a Pro, Premium or Ultimate pass

On a listing page, some cards always lack something: a sold-out product with no price, a promotion that looks like a product. This part reproduces the crash that a loop written for perfect cards hits, then reads the same page so that every gap becomes a recorded None.

In this part:

The loop that crashes on the fourth card
A reader that records what is missing

2 runnable examples (Python) · about 4 minutes

Opening this part with your pass…Pro part. It opens with the same pass as the part above.

Old tutorial code on Beautiful Soup 4.15.0, and the names that go next Pro: opens with a Pro, Premium or Ultimate pass

Much of the scraping code online was written for older releases. This part runs such code on Beautiful Soup 4.15.0 with every warning shown, rewrites it in today’s names, and shows what the old names will do once they are gone.

In this part:

Every warning the old style prints, run
The same scraper in today’s names, warnings as errors
Why the warnings may never reach you
What a removed name will do

4 runnable examples (grid.html, modern_style.py, old_style.py, oldparse.py, quiet_warnings.py, removed_names.py) · about 4 minutes

Opening this part with your pass…Pro part. It opens with the same pass as the part above.

Practise reading cards with Beautiful Soup

Test yourself on parsing, the two kinds of search and turning elements into fields; the first question picks the idea your checkpoint answers left weakest:

Interview questions

Warm-up (fresher): What is the difference between find() and find_all(), and what does each return when nothing matches?

Both search the tree below the tag you call them on, with the same arguments: a tag name, attributes, text or a function. find_all() returns every match, as a list, and an empty list when nothing matches, so a loop over it simply does nothing. find() returns only the first match, and None when there is none, so code that calls a method on its result, such as .get_text(), raises an AttributeError on a page without the element. The CSS pair behaves the same way: select() returns a list, select_one() the first match or None.

Three harder questions start from this lesson: a collector that crashes on a few cards, an old scraper to upgrade, and prices that come out wrong on offer pages. The question page of this module gives the first step of each and the follow-ups to expect:

Fresher to mid

Your Beautiful Soup collector stops with AttributeError: 'NoneType' object has no attribute 'get_text' on 3 of 2,000 product pages. How do you fix it so that it never crashes and never stores a wrong record?

Model answer, Pro. Model answers open with any pass; prices are in the notice on this page. First step and follow-ups
Fresher to mid

You inherit a scraper written years ago with findAll(), findNext() and the text= argument. You want to move from Beautiful Soup 4.15.0 to the next release. How do you make the upgrade safe?

Model answer, Pro. First step and follow-ups
Fresher to mid

Prices read from a shop's offer pages come out as ₹130.00 ₹117.00 for some products and ₹144.00 /kg for others, and a few are glued together without a space. What is going on, and how do you make price extraction reliable?

Model answer, Pro. First step and follow-ups

Key takeaways

BeautifulSoup(html, "html.parser") builds the page’s tree with the parser you name; name one every time, or the tree depends on what is installed.
find_all() and find() match by tag, attributes and text, select() and select_one() by CSS selector; the single-result methods return None when nothing matches.
get_text(" ", strip=True) reads an element’s text as words, tag.get() reads an attribute that may be missing, and urljoin() turns each relative link into an absolute URL.

Terms in this lesson

absolute URL: An address with its scheme and host, such as https://bazaar.nimbu.test/products/NB-0001/. A relative link such as /products/NB-0001/ only has a meaning together with the page it came from.
Beautiful Soup: A Python library that builds a searchable tree from HTML or XML with a parser you choose, and finds elements in it by tag, attributes, text or CSS selector.

References

Beautiful Soup documentation (version 4.15.0) (Beautiful Soup project)
beautifulsoup4 4.15.0 (release files and changelog) (Python Package Index)
Soup Sieve documentation (the CSS selector engine of Beautiful Soup) (Soup Sieve project)
urllib.parse: Parse URLs into components (urljoin), Python 3.14 (Python Software Foundation)
warnings: Warning control (Python 3.14) (Python Software Foundation)
decimal: Decimal fixed-point and floating-point arithmetic (Python 3.14) (Python Software Foundation)

Report a problem with this lesson

In this module: Parsing HTML in Python and JavaScript

HTML, the DOM and DevTools for scrapers Free lesson, Pro answers: 20 minutes
BeautifulSoup 4.15: find, select, get_text Free core, Pro depth: 5 of 15 minutes free
html.parser, lxml or html5lib: why results differ Free core, Pro depth: 6 of 15 minutes free
CSS selectors that survive redesigns Free core, Pro depth: 7 of 20 minutes free
XPath 1.0 in Python: lxml and parsel Free core, Pro depth: 8 of 20 minutes free
Tables, lists and links: clean rows from HTML Free core, Pro depth: 7 of 20 minutes free
Parsing in JavaScript: cheerio and the DOM Free core, Pro depth: 7 of 20 minutes free

Your country

BeautifulSoup 4.15: find, select, get_text

What you will learn

Before you start

What you will learn

Before you start

Beautiful Soup turns the HTML into a tree you can search

Search the tree by tag or by CSS selector

Text, attributes and links become fields

Clean text from messy markup: five traps, measured Pro: opens with a Pro, Premium or Ultimate pass

Pages where fields go missing, read without crashes Pro: opens with a Pro, Premium or Ultimate pass

Old tutorial code on Beautiful Soup 4.15.0, and the names that go next Pro: opens with a Pro, Premium or Ultimate pass

Practise reading cards with Beautiful Soup

Interview questions

Your Beautiful Soup collector stops with AttributeError: 'NoneType' object has no attribute 'get_text' on 3 of 2,000 product pages. How do you fix it so that it never crashes and never stores a wrong record?

You inherit a scraper written years ago with findAll(), findNext() and the text= argument. You want to move from Beautiful Soup 4.15.0 to the next release. How do you make the upgrade safe?

Prices read from a shop's offer pages come out as `₹130.00 ₹117.00` for some products and `₹144.00 /kg` for others, and a few are glued together without a space. What is going on, and how do you make price extraction reliable?

Key takeaways

References

Related tools

What you will learn

Before you start

What you will learn

Before you start

Beautiful Soup turns the HTML into a tree you can search

Search the tree by tag or by CSS selector

Text, attributes and links become fields

Clean text from messy markup: five traps, measured Pro: opens with a Pro, Premium or Ultimate pass

Pages where fields go missing, read without crashes Pro: opens with a Pro, Premium or Ultimate pass

Old tutorial code on Beautiful Soup 4.15.0, and the names that go next Pro: opens with a Pro, Premium or Ultimate pass

Practise reading cards with Beautiful Soup

Interview questions

Your Beautiful Soup collector stops with AttributeError: 'NoneType' object has no attribute 'get_text' on 3 of 2,000 product pages. How do you fix it so that it never crashes and never stores a wrong record?

You inherit a scraper written years ago with findAll(), findNext() and the text= argument. You want to move from Beautiful Soup 4.15.0 to the next release. How do you make the upgrade safe?

Prices read from a shop's offer pages come out as ₹130.00 ₹117.00 for some products and ₹144.00 /kg for others, and a few are glued together without a space. What is going on, and how do you make price extraction reliable?

Key takeaways

References

Related tools

Prices read from a shop's offer pages come out as `₹130.00 ₹117.00` for some products and `₹144.00 /kg` for others, and a few are glued together without a space. What is going on, and how do you make price extraction reliable?