This page needs a newer browser to run its tools. You can still read everything here.

Web Scraping & Browser Automation (Python and JavaScript) Module 1

Permission first: robots.txt, terms, law and limits interview questions

Interview questions on “Permission first: robots.txt, terms, law and limits”: warm-ups, then what each harder question tests and how a good answer starts.

21 questions: 7 warm-ups and 14 with model answers · Levels: Fresher to senior

Warm-up questions

Short questions with short answers, each answered in full.

Fresher

What is the difference between web scraping and web crawling?

Answered in the lesson Collecting web data: API, feed or scraping?
Fresher

What does robots.txt control, and what does it not control?

Answered in the lesson robots.txt for your bot: RFC 9309 rules
Fresher

Why bind a development server to 127.0.0.1 rather than 0.0.0.0?

Answered in the lesson Set up Python, Node and the lab site
Fresher

What is the difference between robots.txt and a site's terms of service?

Answered in the lesson Terms, licences and machine-readable permissions
Fresher

Does India's DPDP Act apply to personal data that is already public?

Answered in the lesson Copyright, data protection and scraping law
Fresher

Why is a SHA-256 hash of a phone number not a safe way to store it?

Answered in the lesson Personal data: collect less, protect the rest
Fresher

Your scraper starts receiving 429 responses. What does that mean, and what do you do?

Answered in the lesson Blocks, CAPTCHAs and login walls: stop

Questions with model answers

What each question tests, the first step of a good answer and the follow-ups to expect.

Fresher to mid

A partner company wants a daily copy of the prices of its own 2,000 products, which its public website shows on listing pages of 24. It asks you to write a scraper for its site. What do you check, and in what order, before the first request?
What it tests

Choosing the first permitted source before writing code, checking permission at each step, and estimating what a collector would cost the site.

First step

Ask the partner for the data in a form made for programs: an export, a feed or access to an API. It is the partner's own data, and a file sent once a day costs its site nothing. Only if that is impossible, look at what the site itself offers, in ladder order: robots.txt and the terms, JSON-LD in the pages, then the listing pages.
Follow-ups
- The partner says the website is the only way. How many requests a day will you send?
- What does your collector send to identify itself?
- A listing page starts answering 429. What changes?
From the lesson Collecting web data: API, feed or scraping?
Model answer, Pro. Model answers open with any pass; prices are in the notice on this page.
Opening the model answers with your pass…
Model answers open with any MySmartCoPilot pass.
Pro pass · 1 week₹79$2.99
30 days₹179$5.99
1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so the model answers can’t be opened at the moment. If you have a pass, enter your licence key.
Choose a pass I have a licence key
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

Your collector fetches about 500 pages an hour from a small community site. Its owner emails that the site has slowed down since your collector started. What do you do in the first hour, and what do you change before it runs again?
What it tests

Taking responsibility for load, stopping first, measuring what was sent from your own logs, and redesigning a collector so that it needs fewer requests.

First step

Pause the collector before you reply, so that the load stops while you talk. Then answer the owner: say what the collector does, how many requests it sent, from where and when, and that it stays paused until you agree a pace. Your own request log gives those numbers.
Follow-ups
- How do you work out how much load you caused?
- How would you collect the same data with fewer requests?
- What will you change so that the owner can reach you next time?
From the lesson Collecting web data: API, feed or scraping?
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

For six hours, every request for a site's robots.txt has answered 503. Your crawler holds a copy of the file from yesterday and 40,000 of the site's addresses in its queue. What does it do now, and what changes if the 503s go on for weeks?
What it tests

Applying RFC 9309's rules for an unreachable robots.txt and a cached copy, and treating a server error as a reason to send less, not more.

First step

Keep yesterday's copy as the rules: RFC 9309 lets a crawler use a cached copy past 24 hours while the file is unreachable. But a 503 for a file as small as robots.txt says the server is in trouble, so pause or sharply slow the page requests to that site, and ask for robots.txt again with a growing interval.
Follow-ups
- How often do you ask for robots.txt again, and when do you resume pages?
- The copy from yesterday may be out of date. Does that matter here?
- What changes after a month of 503s?
From the lesson robots.txt for your bot: RFC 9309 rules
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Fresher to mid

After a refactor, your crawler starts fetching pages under /prices/, which its own group in a site's robots.txt disallows, while every other rule seems to be followed. How do you find the cause, and which test stops it coming back?
What it tests

Debugging a robots.txt decision step by step: the product token, the group chosen, the parser's version and its argument order, and turning known cases into a regression test.

First step

Reproduce one decision outside the crawler: the robots.txt as fetched, one /prices/ address, and the exact agent string the code passes. If that string is a whole User-Agent such as "Mozilla/5.0 (compatible; NimbuBot/1.0)", urllib.robotparser reads only "Mozilla", finds no group and applies the * rules, which say nothing about /prices/.
Follow-ups
- The token is right. What else can make a parser choose the wrong group?
- Which cases go into the regression test?
- How do you stop a parser upgrade from changing decisions without anyone noticing?
From the lesson robots.txt for your bot: RFC 9309 rules
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Fresher to mid

A collector's tests pass on your laptop and fail on the build server, with different values parsed from the same saved page. How do you tell whether the environment is the cause, and what stops it from happening again?
What it tests

Treating versions as part of a result: comparing two environments systematically, pinning exactly, and making the environment visible in every run.

First step

Run the same environment report in both places, the Python version, whether it is a virtual environment, and every installed package with its version, and compare the two line by line. A parser or a Python patch release that differs is the first suspect: the robots.txt lesson's twelve questions get six different answers on Python 3.14.2 and 3.14.8.
Follow-ups
- The reports are identical. What else can differ between the two machines?
- How do you pin so that the build server cannot drift again?
- What should every test run record about its environment?
From the lesson Set up Python, Node and the lab site
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

Every run of your team's build server downloads several hundred megabytes of browsers before the collector's tests start, and builds now take about nine minutes longer. What do you change?
What it tests

Knowing how Playwright, Puppeteer and Selenium get their browsers, downloading only what the tests need, and caching pinned browser builds between runs.

First step

Work out which tool downloads what, and when: Playwright only when playwright install runs (with no browser named, every default browser), Puppeteer from its install script during npm install, Selenium Manager when a test first starts a browser. Then ask what the tests need: runs without a window need one headless Chromium, not three full browsers.
Follow-ups
- How do you keep the browsers between runs without mixing versions?
- The team moves to npm 12 and Chrome stops arriving. Why?
- Which tests, if any, need more than one browser?
From the lesson Set up Python, Node and the lab site
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Fresher to mid

A client wants product reviews from a retailer's site. Its robots.txt allows /reviews/, but its terms forbid automated access. The client says robots.txt is enough. What do you advise, and what could you do instead?
What it tests

Separating the layers of permission, explaining why one yes does not override another layer's no, and finding a legitimate route to the data.

First step

Explain that robots.txt answers only whether a bot may fetch a path, while the terms are the contract for using the site, and they say no to automated access. So the answer for scraping is no, whatever robots.txt allows. Then look for a route the retailer does allow: an API, a feed, a data partnership, or written permission.
Follow-ups
- The client asks how the retailer would ever know. How do you answer?
- The reviews are written by customers. Does that change who can give permission?
- What goes into the permission record for this source?
From the lesson Terms, licences and machine-readable permissions
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

Your team builds a training set for a language model from public web pages. Design the step that decides, for each page, whether it may enter the set, given robots.txt, Content-Signal, Content-Usage, RSL licences and the EU rules on text and data mining.
What it tests

Turning several machine-readable signals with different vocabularies and standing into one conservative decision per page, with evidence that can be checked later.

First step

Decide per page, at fetch time, and record why. Fetch only what robots.txt allows for the crawler's token. Then read every stated preference about training, Content-Signal ai-train, Content-Usage train-ai for the page's path and the RSL licence's usage values, map them to one vocabulary, and let any no exclude the page.
Follow-ups
- Two signals disagree for the same page. Which one wins?
- A site states nothing at all. Is the page in or out?
- How do you handle a site that adds a reservation after you collected its pages?
From the lesson Terms, licences and machine-readable permissions
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

A startup asks you to build a directory of doctors in India from public profile pages: names, clinic phone numbers and qualifications. Before any code is written, which legal questions do you raise, and in what order?
What it tests

Mapping a collection plan to copyright, data protection, access law and the sites' terms, and turning each into a precise question instead of a yes or a no.

First step

Describe the plan by its facts: what it copies, who published each field, how the pages are reached, the purpose, and what each site's terms and robots.txt say. Then ask who made each field public, because the DPDP exclusion covers only data the doctor published herself or that a law requires to be published.
Follow-ups
- Some profiles were written by clinics, not by the doctors. Does that change anything?
- The startup wants anyone who searches to see the phone numbers. What follows from that?
- Which fields would you keep out of the directory entirely?
From the lesson Copyright, data protection and scraping law
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

Your collector has read a site's public price pages for months. The site now blocks it with a 403, and its lawyer writes asking you to stop. A colleague cites hiQ v. LinkedIn and says public pages are fair game. What do you do, and why?
What it tests

Reading access law accurately, knowing the limits of a US preliminary-injunction ruling, and treating a written refusal as a stop under India's IT Act and the site's terms.

First step

Stop the collector for that site at once, keep its state, and reply in writing that it has stopped. Then explain to the colleague that hiQ read one US statute at the preliminary-injunction stage, listed other claims that remain, and binds no Indian court, while the IT Act asks whether the owner gave permission.
Follow-ups
- What do you do with the months of data already collected?
- How could you get the same prices legitimately?
- The site and your company are both outside India. Which questions change?
From the lesson Copyright, data protection and scraping law
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Fresher to mid

Your team scrapes product reviews and wants to keep each reviewer's name "to remove duplicate reviews". How do you design the data handling so that deduplication works without storing the names?
What it tests

Applying minimisation and keyed pseudonyms to a real requirement, and asking first whether a field is needed at all.

First step

Ask what deduplication actually needs. A duplicate is usually the same text posted twice, so a keyed hash of the normalised review text may be enough. If the same author matters too, a keyed pseudonym of the name, computed before storage, links reviews without storing the name, and it gets a deletion date like any personal field.
Follow-ups
- Two reviewers share a name. Does the pseudonym still work?
- Where does the key live, and who may use it?
- The legal team asks how long you keep the pseudonyms. What do you answer?
From the lesson Personal data: collect less, protect the rest
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

A dataset of 200,000 scraped seller records leaks. Its phone numbers were stored as plain SHA-256 hashes. How exposed are the sellers, and what do you do in the first day?
What it tests

Recognising that unkeyed hashes of a small set of identifiers can be reversed, treating the leak as a personal data breach, and changing the design.

First step

Treat the numbers as leaked in plain text: ten-digit numbers have at most ten billion values, and anyone can check each hash against them. That makes the leak a personal data breach. Contain it, record what leaked and since when, and start the notifications the law requires where you operate.
Follow-ups
- Which law tells you whom to notify, and how fast?
- How do you store the numbers from now on, if at all?
- What else in the records could identify the sellers?
From the lesson Personal data: collect less, protect the rest
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

A nightly crawl of 40 shops must stop for any shop that blocks it, keep going for the others, and resume the next night without losing work. How do you design the stopping and the saved state?
What it tests

Designing per-host circuit breakers, telling stop signals from soft ones, saving state for a person, and resuming without repeating blocked requests.

First step

Give each host its own breaker and queue, and classify every response first. Logins, CAPTCHAs, challenges and refusals open the host's breaker at once; 429s, 503s and unexpected pages are soft signs, and three in a row open it. An open breaker sends nothing more, writes its reason and queue to durable state, and waits for a person.
Follow-ups
- How does the next night's run treat a host whose breaker is still open?
- One shop blocks a single path only. One breaker per host or per path?
- How do you notice a block page your classifier does not know yet?
From the lesson Blocks, CAPTCHAs and login walls: stop
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Fresher to mid

A site you collect prices from adds a CAPTCHA. Your manager asks you to plug in a CAPTCHA-solving service so that the dashboard keeps working. What do you answer, and what do you propose instead?
What it tests

Explaining why a block is a refusal rather than a technical obstacle, the risks of getting round one, and finding a legitimate route to the data.

First step

Say no clearly, with the reasons: the CAPTCHA is the site's decision that programs should not collect there, getting round an access control is what access laws ask about, and it defeats a safeguard regulators expect sites to keep. Then offer a plan that keeps the dashboard honest while you look for a permitted source.
Follow-ups
- The manager says competitors do it. How do you respond?
- Which legitimate routes do you try first, and in what order?
- What does the dashboard show while you wait for the site's answer?
From the lesson Blocks, CAPTCHAs and login walls: stop
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes

The lessons of this module

Collecting web data: API, feed or scraping? Free lesson, Pro answers: 25 minutes
Set up Python, Node and the lab site Free lesson, Pro answers: 20 minutes
robots.txt for your bot: RFC 9309 rules Free lesson, Pro answers: 30 minutes
Terms, licences and machine-readable permissions Free lesson, Pro answers: 20 minutes
Copyright, data protection and scraping law Free lesson, Pro answers: 25 minutes
Personal data: collect less, protect the rest Free lesson, Pro answers: 20 minutes
Blocks, CAPTCHAs and login walls: stop Free lesson, Pro answers: 25 minutes

All Web Scraping & Browser Automation (Python and JavaScript) interview questions · The Web Scraping & Browser Automation (Python and JavaScript) track

Warm-up questions

What is the difference between web scraping and web crawling?

What does robots.txt control, and what does it not control?

Why bind a development server to 127.0.0.1 rather than 0.0.0.0?

What is the difference between robots.txt and a site's terms of service?

Does India's DPDP Act apply to personal data that is already public?

Why is a SHA-256 hash of a phone number not a safe way to store it?

Your scraper starts receiving 429 responses. What does that mean, and what do you do?

Questions with model answers

A partner company wants a daily copy of the prices of its own 2,000 products, which its public website shows on listing pages of 24. It asks you to write a scraper for its site. What do you check, and in what order, before the first request?

Your collector fetches about 500 pages an hour from a small community site. Its owner emails that the site has slowed down since your collector started. What do you do in the first hour, and what do you change before it runs again?

For six hours, every request for a site's robots.txt has answered 503. Your crawler holds a copy of the file from yesterday and 40,000 of the site's addresses in its queue. What does it do now, and what changes if the 503s go on for weeks?

After a refactor, your crawler starts fetching pages under /prices/, which its own group in a site's robots.txt disallows, while every other rule seems to be followed. How do you find the cause, and which test stops it coming back?

A collector's tests pass on your laptop and fail on the build server, with different values parsed from the same saved page. How do you tell whether the environment is the cause, and what stops it from happening again?

Every run of your team's build server downloads several hundred megabytes of browsers before the collector's tests start, and builds now take about nine minutes longer. What do you change?

A client wants product reviews from a retailer's site. Its robots.txt allows /reviews/, but its terms forbid automated access. The client says robots.txt is enough. What do you advise, and what could you do instead?

Your team builds a training set for a language model from public web pages. Design the step that decides, for each page, whether it may enter the set, given robots.txt, Content-Signal, Content-Usage, RSL licences and the EU rules on text and data mining.

A startup asks you to build a directory of doctors in India from public profile pages: names, clinic phone numbers and qualifications. Before any code is written, which legal questions do you raise, and in what order?

Your collector has read a site's public price pages for months. The site now blocks it with a 403, and its lawyer writes asking you to stop. A colleague cites hiQ v. LinkedIn and says public pages are fair game. What do you do, and why?

Your team scrapes product reviews and wants to keep each reviewer's name "to remove duplicate reviews". How do you design the data handling so that deduplication works without storing the names?

A dataset of 200,000 scraped seller records leaks. Its phone numbers were stored as plain SHA-256 hashes. How exposed are the sellers, and what do you do in the first day?

A nightly crawl of 40 shops must stop for any shop that blocks it, keep going for the others, and resume the next night without losing work. How do you design the stopping and the saved state?

A site you collect prices from adds a CAPTCHA. Your manager asks you to plug in a CAPTCHA-solving service so that the dashboard keeps working. What do you answer, and what do you propose instead?