Web Scraping & Browser Automation (Python and JavaScript)
Web Scraping & Browser Automation (Python and JavaScript) practice
43 exercises and 38 quizzes from the lessons of this track. Sample tests run in your browser; what you pass is kept in this browser only.
Permission first: robots.txt, terms, law and limits
- Choose the first permitted source for twelve data needs
- Quiz: Collecting web data: API, feed or scraping?
- Read a lab check report and list what to fix
- Quiz: Set up Python, Node and the lab site
- Turn the robots.txt answer into a decision
- Find the robots.txt rule that decides a path
- Quiz: robots.txt for your bot: RFC 9309 rules
- Combine Content-Signal and Content-Usage into one answer
- Quiz: Terms, licences and machine-readable permissions
- Decide whether the DPDP Act applies to a field
- Quiz: Copyright, data protection and scraping law
- Keep, mask, pseudonymise or drop each field
- Quiz: Personal data: collect less, protect the rest
- Classify a response as continue, skip, wait or stop
- Give every host a breaker that stops the crawl
- Quiz: Blocks, CAPTCHAs and login walls: stop
HTTP, sessions, pagination and APIs
- Decide what to do with each answer before parsing it
- Quiz: HTTP for scrapers: requests, responses, headers
- Build the request a browser would send for a form
- Quiz: Sessions, cookies and HTML forms
- Page through any listing with four ways out
- Quiz: Pagination: pages, next links, cursors, Link
- Decode a page in the order the HTML standard uses
- Quiz: Encodings: getting Hindi text and ₹ right
- Decide how long to wait after an API's answer
- Quiz: Use the API first: JSON, keys and quotas
- Find the pages a sitemap says have changed
- Quiz: Feeds, sitemaps and JSON-LD before HTML
Parsing HTML in Python and JavaScript
- Write the path from the root to a piece of text
- Quiz: HTML, the DOM and DevTools for scrapers
- Turn a price element's text into whole paise
- Quiz: BeautifulSoup 4.15: find, select, get_text
- Quiz: html.parser, lxml or html5lib: why results differ
- Raise an alert when a selector's matches fall
- Quiz: CSS selectors that survive redesigns
- Quote any label as an XPath 1.0 string
- Quiz: XPath 1.0 in Python: lxml and parsel
- Expand spans by the table rules browsers follow
- Quiz: Tables, lists and links: clean rows from HTML
- Read a table's rows from either parser's tree
- Quiz: Parsing in JavaScript: cheerio and the DOM
Dynamic pages and browser automation
Politeness, caching and robust crawls
- A client that paces itself and retries within limits
- Decide how long to wait before a retry, or not to retry
- Quiz: Rate limits, 429 and Retry-After
- Fetch a page only when it has changed
- Quiz: Caching and conditional requests: ETag, 304
- Plan a crawl that is polite to every host at once
- Quiz: Concurrency that stays polite: asyncio and Node
- Build the signature base of a Web Bot Auth request
- Quiz: Identify your bot: User-Agent and Web Bot Auth
- Guard a crawl against session addresses and endless variants
- Quiz: Resumable crawls, crawl traps and safe links
Scrapy 2.19 spiders, pipelines and operations
- Follow a page's links the way response.follow and the duplicate filter do
- Check a crawl's settings before it starts
- Quiz: Scrapy 2.19: project, spider and async start()
- Write a pipeline that validates, deduplicates and upserts into SQLite
- Quiz: Scrapy items, loaders and pipelines
- Audit a crawl's access log for Crawl-delay and Retry-After
- Quiz: Scrapy politeness: robots, throttle, cache
- Keep the sitemap entries that changed since a moment
- Quiz: CrawlSpider, SitemapSpider and link rules
Cleaning, validating and storing data
- Read the board's dates day first, with the offset
- Turn a price text into whole paise
- Bring every price to paise per kilogram
- Quiz: Clean prices, dates, units and Indian numbers
- Validate a run, quarantine its failures and measure it
- Quiz: Validate every record: schemas and quarantine
- A store that a crawl can run into any number of times
- Quiz: Store and dedupe: SQLite, JSON Lines, Parquet
- A change detector that ignores noise and confirms the implausible
- Quiz: Detect changes over time: hashes and diffs
Scheduling, monitoring, testing and safety
Office automation: Excel, PDF and email
- The rows and totals of the weekly report, exact to the paisa
- Quiz: Excel automation: openpyxl, XlsxWriter, ExcelJS
- Turn a board's pages into checked records
- Quiz: PDFs: extract text and tables, merge, generate
- Sort the answers to a report before anyone reads them
- Quiz: Email reports: build, send and read safely
Results here are what this browser has checked: feedback for you, not proof. They stay on this device; your Learn progress can be exported to a file. · Back to the Web Scraping & Browser Automation (Python and JavaScript) track