This page needs a newer browser to run its tools. You can still read everything here.

Data Analysis with Python (pandas, polars, DuckDB) Module 9

Text extraction and categorical data interview questions

Interview questions on text extraction and categorical data: warm-ups, then what each harder question tests and how a good answer starts.

8 questions: 3 warm-ups and 5 with model answers · Levels: Fresher to senior

Warm-up questions

Short questions with short answers, each answered in full.

Fresher

What is the difference between str.extract and str.extractall in pandas?

Answered in the lesson Extracting fields from text with regular expressions
Fresher to mid

Why does df["code"].str.contains("2.5") match OIL-235L, and how do you search for the text itself?

Answered in the lesson Extracting fields from text with regular expressions
Fresher

What does the pandas category dtype store, and when does converting a text column to it save memory?

Answered in the lesson Categorical and ordered data

Questions with model answers

What each question tests, the first step of a good answer and the follow-ups to expect.

Fresher to mid

You extracted PIN codes from 2 million typed delivery addresses, and 7 % of the rows came back without one. How do you decide what to repair automatically, what to send back, and what to report?
What it tests

Treating extraction failures as data: sampling and classifying them, telling safe repairs from guesses, validating every repaired value, and reporting counts that reconcile with the input.

First step

Leave the pattern alone at first. Draw a random sample of about 400 failed rows, label why each one failed (no PIN written, a space inside it, letters for digits, another script's digits, a placeholder such as 000000), and count the labels. The counts show which repairs are worth automating and how many rows each would recover.
Follow-ups
- A repair would recover 2 % of all rows but guesses wrong on some of them. Do you ship it?
- How do you make sure a new pattern does not break rows that worked before?
- The business wants one number, deliveries with a usable PIN code. What do you report?
From the lesson Extracting fields from text with regular expressions
Model answer, Pro. Model answers open with any pass; prices are in the notice on this page.
Opening the model answers with your pass…
Model answers open with any MySmartCoPilot pass.
Pro pass · 1 week₹79$2.99
30 days₹179$5.99
1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so the model answers can’t be opened at the moment. If you have a pass, enter your licence key.
Choose a pass I have a licence key
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 4 minutes
Mid to senior

A phone-number regex gives one count in your pandas notebook, a smaller count in the DuckDB pipeline, and an error in polars. Walk through what you check and how you make the three agree.
What it tests

Knowing that pandas, polars and DuckDB use different regex engines and defaults, reproducing a difference on a few rows, and turning one pattern into a shared, tested definition.

First step

Run the pattern in each engine on a handful of rows you can check by eye: a plain number, one with +91, one with spaces, one in another script's digits and a near miss. Then compare the engines on three points: which regex library each uses, whether \d means 0 to 9 or any digit, and whether lookarounds are allowed.
Follow-ups
- The pattern uses a lookbehind. How do you rewrite it?
- DuckDB returns an empty string where pandas returns NaN. Which numbers does that change?
- How do you stop the three versions drifting apart again?
From the lesson Extracting fields from text with regular expressions
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 4 minutes
Fresher to mid

After a survey's rating column was converted to an ordered category, the dashboard shows 3 % fewer Excellent answers than the raw export. Where did the answers go, and how do you change the pipeline so that this cannot happen silently again?
What it tests

Knowing that a conversion to a category list turns unknown labels into missing values, finding those labels, and putting a guard and a test around the conversion.

First step

Compare the raw labels with the category list before anything else: count every raw value that is not exactly one of the categories, grouped by its spelling. Variants of Excellent, extra spaces and answers the form added later show up at once, and their counts should add up to the gap.
Follow-ups
- Which normalisations are safe to automate, and which need a person?
- The form adds a new answer, Very good, next month. What should happen?
- How would the same pipeline behave in polars with an Enum?
From the lesson Categorical and ordered data
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

A weekly report groups sales by store, product and week, all categorical, with observed=False so that empty combinations show as zero. It used to run in seconds; since the product list grew it runs for minutes and then out of memory. What happened, and how do you rebuild the report?
What it tests

Understanding that observed=False builds every combination of the categories, estimating its size before running, and producing complete reports without that product.

First step

Multiply the category counts before running anything: 400 stores × 3,000 products × 52 weeks is 62.4 million groups, whatever the number of sales rows. Compare that with the combinations that actually occur, len(df[keys].drop_duplicates()), which can never exceed the number of rows.
Follow-ups
- The business wants a zero for every product a store stocks but did not sell. How do you produce it?
- Does an upgrade from pandas 2 to pandas 3 change what this code returns?
- How would you write the same complete report in polars or SQL?
From the lesson Categorical and ordered data
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

Each monthly file loads in 400 MB because its text columns are categories. After pd.concat of twelve files the DataFrame needs 12 GB instead of the expected 4.8 GB, and its text columns show the dtype str. Explain what happened and how you would load the year.
What it tests

Knowing how concat and merge treat category columns whose lists differ, measuring the cost, and keeping one category definition across files and through storage.

First step

Compare the category lists of two files, for example df["city"].cat.categories, before suspecting anything else. When a list differs by even one label, concat falls back to the labels' own dtype, str, and the memory saved by that column is gone.
Follow-ups
- union_categoricals or one shared CategoricalDtype, which do you choose?
- A merge on a categorical key returns a str key. Why, and does it matter?
- How would you store the year so the next load is fast and keeps the categories?
From the lesson Categorical and ordered data
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes

All Data Analysis with Python (pandas, polars, DuckDB) interview questions · The Data Analysis with Python (pandas, polars, DuckDB) track

Warm-up questions

What is the difference between str.extract and str.extractall in pandas?

Why does df["code"].str.contains("2.5") match OIL-235L, and how do you search for the text itself?

What does the pandas category dtype store, and when does converting a text column to it save memory?

Questions with model answers

You extracted PIN codes from 2 million typed delivery addresses, and 7 % of the rows came back without one. How do you decide what to repair automatically, what to send back, and what to report?

A phone-number regex gives one count in your pandas notebook, a smaller count in the DuckDB pipeline, and an error in polars. Walk through what you check and how you make the three agree.

After a survey's rating column was converted to an ordered category, the dashboard shows 3 % fewer Excellent answers than the raw export. Where did the answers go, and how do you change the pipeline so that this cannot happen silently again?

A weekly report groups sales by store, product and week, all categorical, with observed=False so that empty combinations show as zero. It used to run in seconds; since the product list grew it runs for minutes and then out of memory. What happened, and how do you rebuild the report?

Each monthly file loads in 400 MB because its text columns are categories. After pd.concat of twelve files the DataFrame needs 12 GB instead of the expected 4.8 GB, and its text columns show the dtype str. Explain what happened and how you would load the year.