This page needs a newer browser to run its tools. You can still read everything here.

Web Scraping & Browser Automation (Python and JavaScript) Module 7

Cleaning, validating and storing data interview questions

Interview questions on cleaning, validating and storing data: warm-ups, then what each harder question tests and how a good answer starts.

15 questions: 4 warm-ups and 11 with model answers · Levels: Fresher to senior

Warm-up questions

Short questions with short answers, each answered in full.

Fresher

Why should a scraper store prices as whole paise, or as Decimal, and never as float?

Answered in the lesson Clean prices, dates, units and Indian numbers
Fresher

What is the difference between cleaning a scraped value and validating a record?

Answered in the lesson Validate every record: schemas and quarantine
Fresher

What is an upsert, and why does a scraper need one?

Answered in the lesson Store and dedupe: SQLite, JSON Lines, Parquet
Fresher

Why hash a normalised record, rather than the raw page, to detect changes?

Answered in the lesson Detect changes over time: hashes and diffs

Questions with model answers

What each question tests, the first step of a good answer and the follow-ups to expect.

Fresher to mid

A price tracker built on scraped product pages shows weekly totals that differ from the shop's own by a few paise, and a handful of spice prices that are 100 times too low. How do you find the causes, and what do you change?
What it tests

Recognising float arithmetic on money, reading the raw text behind a wrong number, and repairing stored prices from kept text instead of guessing.

First step

Put each disagreeing row beside the text it was cleaned from. A total off by a paisa or two points at float arithmetic somewhere between the text and the store, so search the pipeline for float(). A value 100 times too low points at the reading of the text: the dot of Rs. taken as a decimal point, or rupees stored as if they were paise.
Follow-ups
- Some products are sold by the quintal on one page and by the gram on another. How do you compare them?
- How do you repair a year of stored prices without fetching every page again?
- Which tests would have caught both problems before release?
From the lesson Clean prices, dates, units and Indian numbers
Model answer, Pro. Model answers open with any pass; prices are in the notice on this page.
Opening the model answers with your pass…
Model answers open with any MySmartCoPilot pass.
Pro pass · 1 week₹79$2.99
30 days₹179$5.99
1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so the model answers can’t be opened at the moment. If you have a pass, enter your licence key.
Choose a pass I have a licence key
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Fresher to mid

A dashboard built from scraped bulletins shows sharp peaks of activity on the first twelve days of some months and almost nothing on the same days of others. What do you suspect, how do you confirm it, and how do you repair the data?
What it tests

Recognising a day and month swap from a month-first reading, confirming it from the raw text and the error log, and fixing the pipeline with one explicit format per source.

First step

Count the stored dates by source and by day of the month. A month-first reading of DD-MM-YYYY text moves every day from 1 to 12 into another month without an error and refuses days 13 to 31, so a source with nothing stored on days 13 and later, or an error log full of them, confirms the swap.
Follow-ups
- The source writes two-digit years. What else can go wrong?
- Some rows were stored before the fix and some after. How do you tell them apart?
- How do you stop the next new source from repeating this?
From the lesson Clean prices, dates, units and Indian numbers
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

Design the quarantine for a collector that stores about 50,000 product records a night: what each row holds, how a person works through it, and how fixed records reach the main table without duplicates.
What it tests

Keeping failures as evidence with the raw input and every reason, review by groups of reasons, idempotent replay by natural key, and versioned rules.

First step

Start from what a person needs to act on one failure: the raw item as the parser returned it, the page address, the run, the version of the rules and every reason. Then size it: at a 1 % failure rate that is 500 rows a night, so the review works on rows grouped by reason, never one at a time.
Follow-ups
- How long do you keep quarantined rows?
- A replay moved 3,000 rows, and now a report counts some products twice. What went wrong?
- How do you notice a quarantine that is quietly growing?
From the lesson Validate every record: schemas and quarantine
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Fresher to mid

After a JavaScript collector moved to a schema library with coercion switched on, about 3 % of products show a price of 0, and validation passes them. What happened, and how do you change the schema?
What it tests

Knowing how Number() converts empty and odd strings, why a coercing schema inherits those rules, and where conversion belongs in a collector.

First step

Look at the raw text of the zero-priced products before changing anything. If their price cells are empty or hold only spaces, work out what the schema's conversion makes of such a string, and whether the schema's checks run before or after that conversion.
Follow-ups
- Why did the old hand-written check catch these rows?
- Which other inputs does Number() turn into plausible numbers?
- How do you repair the products already stored with a price of 0?
From the lesson Validate every record: schemas and quarantine
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

Every record of last night's run passed validation, but the average price per kilogram across the catalogue fell by half. How would you have caught it before publishing, and which thresholds would you set?
What it tests

Checks across a whole run rather than per record: counts, the share of changed values and the size of the changes against the last published run, with limits set from history.

First step

Ask what changed between nights, not within one: line up the night's prices with those of a night you trust, product by product, and look at how many changed and by how much. A sale moves some prices by a little; a change in the page moves many of them by a lot, and in the same direction.
Follow-ups
- Real sales also change many prices. How do you avoid holding every sale night?
- Which night do you compare with after a night was held?
- How do you catch a lost category that is only 2 % of the catalogue?
From the lesson Validate every record: schemas and quarantine
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Fresher to mid

A crawl was restarted after a crash halfway through, and the daily report now counts about half of the products twice. How do you make the store safe to re-run, and how do you clean up the rows already doubled?
What it tests

Natural keys, upserts and transactions for a store that can be re-run, and a clean-up of duplicates that keeps the right values.

First step

Count rows per product key: the products seen before the crash will have two rows each. Then read the table's definition. A table whose only unique column is an automatic id has nothing that says two rows are the same product, so every sighting becomes a new row.
Follow-ups
- The two rows of a product have different prices. Which one do you keep?
- How do you make sure that a crash halfway leaves no half-written crawl?
- How would a test catch this before release?
From the lesson Store and dedupe: SQLite, JSON Lines, Parquet
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

You must keep two years of daily prices for 50,000 products and answer what any product cost on any day. Where do you keep current records, raw pages and the price history, and what does each choice cost?
What it tests

Matching stores to access patterns, sizing a history as changes rather than daily copies, and keeping raw pages small.

First step

Size it first. A full copy every day is 50,000 × 730, about 36.5 million rows; if 2 % of prices change on a typical day, a history of changes is about 1,000 rows a day, 730,000 in two years. Then choose a store for each pattern: current records change, history only grows, raw pages are written once.
Follow-ups
- How do you answer the price on a given day from a history of changes?
- Raw pages are about 2.5 KB each. How much space do two years take, and how do you shrink it?
- Which part would you keep in Parquet, and when would you not?
From the lesson Store and dedupe: SQLite, JSON Lines, Parquet
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Fresher to mid

A partner opened your CSV export of scraped reviews in a spreadsheet app, and one review ran as a formula. What went wrong, and how do you export safely without corrupting values such as negative price changes?
What it tests

Recognising formula characters at the start of text cells, neutralising text columns only, and testing every export.

First step

Find the review in the file: it begins with a character that a spreadsheet app reads as the start of a formula, such as an equals sign. A CSV has no types, so the app decided what the cell was. The fix belongs in the export, which wrote text from pages into a file that people open.
Follow-ups
- Why not prefix every cell that starts with a minus sign?
- The partner saves the file and opens it again. Is it still safe?
- How do you test the export?
From the lesson Store and dedupe: SQLite, JSON Lines, Parquet
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

Your price monitor sends about 300 change alerts every morning for 2,000 watched pages, and nearly all of them are noise. How do you bring the alerts down to real changes without missing the real ones?
What it tests

Separating noise from data with hashes of normalised records, field-level diffs, unknown values and confirmation, and measuring the result.

First step

Sample a morning's alerts and sort them by what changed on the page: a token or a time, a banner, a missing field, or a field that really changed. Each kind has its own fix, and the counts say which fix to make first.
Follow-ups
- How do you know that the fix did not hide real changes?
- Some pages serve two templates at once. What changes?
- How do you notice that a field has disappeared for good?
From the lesson Detect changes over time: hashes and diffs
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Fresher to mid

A watched product's price fell by 99 % overnight, and your system alerted 5,000 subscribers. By morning the site showed the old price again. What should the detector have done, step by step?
What it tests

Plausibility checks, confirmation with a second fetch, holding the largest jumps for a person, and sending only confirmed values.

First step

Look at what the detector saw: one fetch, one reading, and a drop far beyond anything in the product's history. A change that large is implausible, and an implausible reading should be confirmed before anyone acts on it.
Follow-ups
- The second fetch, five minutes later, showed the same 99 % drop. Now what?
- How do you correct the alert to 5,000 subscribers without making it worse?
- Where else should the same rule apply?
From the lesson Detect changes over time: hashes and diffs
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

A month after your monitor told subscribers about a price cut, the shop says its price never changed. How do you show what its page said, and what should your system keep so that you always can?
What it tests

Keeping evidence: raw fetches with digests, a history with valid_from and valid_to, and change events tied to the fetches behind them.

First step

Find the change event and the fetch it came from: the address, the time and the stored page. If the stored page shows the cut and its digest checks, you have the evidence; if nothing but the number was kept, you have only your word against the shop's.
Follow-ups
- How do you show that the stored page was not edited later?
- How much does keeping every fetch cost, and how do you reduce it?
- What if the page differs for visitors in different places?
From the lesson Detect changes over time: hashes and diffs
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes

The lessons of this module

Clean prices, dates, units and Indian numbers Free lesson, Pro answers: 30 minutes
Validate every record: schemas and quarantine Free core, Pro depth: 7 of 20 minutes free
Store and dedupe: SQLite, JSON Lines, Parquet Free core, Pro depth: 8 of 20 minutes free
Detect changes over time: hashes and diffs Free core, Pro depth: 7 of 20 minutes free

All Web Scraping & Browser Automation (Python and JavaScript) interview questions · The Web Scraping & Browser Automation (Python and JavaScript) track

Warm-up questions

Why should a scraper store prices as whole paise, or as Decimal, and never as float?

What is the difference between cleaning a scraped value and validating a record?

What is an upsert, and why does a scraper need one?

Why hash a normalised record, rather than the raw page, to detect changes?

Questions with model answers

A price tracker built on scraped product pages shows weekly totals that differ from the shop's own by a few paise, and a handful of spice prices that are 100 times too low. How do you find the causes, and what do you change?

A dashboard built from scraped bulletins shows sharp peaks of activity on the first twelve days of some months and almost nothing on the same days of others. What do you suspect, how do you confirm it, and how do you repair the data?

Design the quarantine for a collector that stores about 50,000 product records a night: what each row holds, how a person works through it, and how fixed records reach the main table without duplicates.

After a JavaScript collector moved to a schema library with coercion switched on, about 3 % of products show a price of 0, and validation passes them. What happened, and how do you change the schema?

Every record of last night's run passed validation, but the average price per kilogram across the catalogue fell by half. How would you have caught it before publishing, and which thresholds would you set?

A crawl was restarted after a crash halfway through, and the daily report now counts about half of the products twice. How do you make the store safe to re-run, and how do you clean up the rows already doubled?

You must keep two years of daily prices for 50,000 products and answer what any product cost on any day. Where do you keep current records, raw pages and the price history, and what does each choice cost?

A partner opened your CSV export of scraped reviews in a spreadsheet app, and one review ran as a formula. What went wrong, and how do you export safely without corrupting values such as negative price changes?

Your price monitor sends about 300 change alerts every morning for 2,000 watched pages, and nearly all of them are noise. How do you bring the alerts down to real changes without missing the real ones?

A watched product's price fell by 99 % overnight, and your system alerted 5,000 subscribers. By morning the site showed the old price again. What should the detector have done, step by step?

A month after your monitor told subscribers about a price cut, the shop says its price never changed. How do you show what its page said, and what should your system keep so that you always can?