This page needs a newer browser to run its tools. You can still read everything here.

Data Analysis with Python (pandas, polars, DuckDB) Module 14

Performance and scale interview questions

Interview questions on performance and scale: short warm-ups, then what each harder question tests, the first step of a good answer and its follow-ups.

11 questions: 3 warm-ups and 8 with model answers · Levels: Fresher to staff

Warm-up questions

Short questions with short answers, each answered in full.

Fresher

Why is a vectorised pandas expression faster than the same calculation in apply(axis=1)?

Answered in the lesson Fast pandas: vectorise and use the right dtypes
Fresher to mid

You read a large CSV in chunks and compute each chunk's mean order value. Why is the average of those chunk means usually wrong, and how do you get the exact mean?

Answered in the lesson Bigger than memory: chunks, out-of-core, samples
Fresher

What is Apache Arrow, and why does it matter when one workflow uses pandas, polars and DuckDB?

Answered in the lesson pandas, polars or DuckDB: a measured guide

Questions with model answers

What each question tests, the first step of a good answer and the follow-ups to expect.

Fresher to mid

A pricing step runs df.apply(price_row, axis=1) over 4 million order lines. price_row is an if/elif chain on the customer tier, the quantity and the product category, and the step takes six minutes. How do you rewrite it, and how do you prove that no price changed?
What it tests

Turning row-wise branching into vectorised conditions in the right order, handling boundaries and missing values, and proving that two implementations agree before switching.

First step

Write the branches down as a table of conditions in the order the if/elif chain checks them, then express them with np.select, which also takes the first condition that is true. Keep the old function: it becomes the reference the new version must match on a large random sample and on edge cases made by hand.
Follow-ups
- One branch calls a discount service for every customer. What do you do with it?
- The new prices differ from the old ones in the tenth decimal place. Is that a bug?
- How do you keep a later rule change from breaking the equivalence?
From the lesson Fast pandas: vectorise and use the right dtypes
Model answer, Pro. Model answers open with any pass; prices are in the notice on this page.
Opening the model answers with your pass…
Model answers open with any MySmartCoPilot pass.
Pro pass · 1 week₹79$2.99
30 days₹179$5.99
1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so the model answers can’t be opened at the moment. If you have a pass, enter your licence key.
Choose a pass I have a licence key
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 4 minutes
Mid to senior

df.memory_usage(deep=True).sum() reports 1.2 GB, but the Python process peaks at 4.5 GB while it loads the CSV and stays at 3 GB afterwards. Explain the gap, and how you bring the peak down.
What it tests

What memory_usage counts and what it leaves out (parser buffers, temporary copies, other objects, freed memory the process keeps), measuring a peak, and lowering it with columns, dtypes and chunks.

First step

Measure before changing anything: record the process's peak memory around read_csv alone, then around each later step. memory_usage counts only the finished frame's columns and index; the parser's buffers, intermediate copies and memory that was freed but not handed back to the operating system never appear in it.
Follow-ups
- Which read_csv arguments lower the peak most?
- Memory stays at 3 GB after del df. Is that a leak?
- When do you stop tuning pandas and switch to chunks or another engine?
From the lesson Fast pandas: vectorise and use the right dtypes
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 4 minutes
Mid to senior

You need the exact median and 95th percentile of order value per city from a 60 GB CSV on a 16 GB laptop. A colleague's chunked pandas script takes the median of the chunk medians. What is wrong with it, and what do you do instead?
What it tests

Knowing which statistics combine across chunks, computing exact quantiles on data bigger than memory, and judging when an approximate quantile with a stated error is acceptable.

First step

Explain the bug first: a median depends on the position of every value, so chunk medians cannot be combined into it, and the error changes with the order of the file. Then ask whether the answer must be exact, because exact and approximate quantiles call for different tools and different amounts of memory.
Follow-ups
- How would you compute it exactly without DuckDB or polars?
- The business accepts an error of 1 %. What changes?
- How much memory does each city need, and which city decides it?
From the lesson Bigger than memory: chunks, out-of-core, samples
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 4 minutes
Senior to staff

A DuckDB job joins a 300 GB Parquet event table to a 40 GB user table and aggregates by country. On a 64 GB machine shared with other services it fails with an Out of Memory Error, and on some nights the process is simply killed. What do you change, and in what order?
What it tests

How DuckDB's memory limit, threads, temporary directory and insertion order interact, shrinking the data before the join, and telling a DuckDB error from the operating system killing the process.

First step

Separate the two symptoms. An Out of Memory Error is DuckDB refusing to pass its memory limit; a killed process means the limit is higher than the memory the machine can really give. Record memory_limit, threads, temp_directory and the process's peak memory before changing anything.
Follow-ups
- The temporary directory fills the disk. What now?
- Which part of the query do you rewrite first?
- How do you make the job dependable as a nightly run?
From the lesson Bigger than memory: chunks, out-of-core, samples
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Fresher to mid

An analyst's notebook, built on a 1 % sample of orders, reports that sales in a small city grew 40 % from one month to the next. On the full data the growth was 6 %. How does that happen, and how do you set up sample-based exploration so that it cannot mislead a report?
What it tests

Sampling error for small groups, comparing two noisy estimates, seeds and repeatability, stratified samples, and the rule that reported numbers come from the full data.

First step

Count the sample rows behind the number. A city with 2,000 orders a month has about 20 of them in a 1 % sample, so each month's total rests on about 20 orders, and the ratio of two such totals is noisier still. Then recompute both months on the full data before anyone discusses causes.
Follow-ups
- How large would the sample need to be for that city?
- Drawn again with another seed, the sample shows a fall of 15 %. What does that tell you?
- Which numbers may a report take from a sample?
From the lesson Bigger than memory: chunks, out-of-core, samples
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

A colleague's notebook shows polars running 10 times faster than pandas on "our workload" and proposes moving every report to polars. What do you check before agreeing, and how would you run a fair comparison?
What it tests

Reading a benchmark critically (tasks, data, hardware, versions, warm-up, what is timed, identical results) and weighing a speed-up against where the time really goes and what a migration costs.

First step

Ask what exactly was timed: which tasks, how many rows and which column types, on which machine, whether imports and file reading were included, how many runs, and whether both versions returned the same result. Then profile the real reports, because a ratio on one task says little about a job that mostly waits for a database.
Follow-ups
- The 10 times comes from reading CSV files. Does that change your answer?
- How do you prove that the polars reports return the same numbers?
- What would make you say no even though polars is faster?
From the lesson pandas, polars or DuckDB: a measured guide
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

After DuckDB is added to a pandas 3 pipeline, a query on a 1,000,000-row DataFrame takes about 0.2 seconds per call instead of milliseconds, inside a loop that runs 2,000 times. What is happening, and how do you fix it?
What it tests

Conversion costs at engine boundaries, measuring them apart from the query, and restructuring code so that data crosses a boundary once.

First step

Time the same query on a Parquet copy of the data, and on a frame without its text columns. If those are fast, the time goes into DuckDB taking in the DataFrame on every call, not into the query itself, and the loop pays that cost 2,000 times: about 400 seconds.
Follow-ups
- Why can a text column slow a query that never uses it?
- How do you rewrite the loop?
- Would handing DuckDB a polars DataFrame instead help?
From the lesson pandas, polars or DuckDB: a measured guide
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Senior to staff

Your analytics team of eight writes pandas notebooks, and data volumes have grown tenfold in a year. Propose a default engine and a policy for exceptions, and say how you would know after six months whether the change worked.
What it tests

Turning measurements into a team decision: profiling the workload, the cost of migrating, fit with the team's skills and libraries, a written decision record, and measures of success.

First step

Measure before choosing: list the team's recurring jobs with their input sizes, run times and peak memory, and find where the time and the memory go. Then try the two or three heaviest jobs in each candidate engine on production-like hardware, with the outputs checked for equality.
Follow-ups
- Half the team does not know SQL. Does that rule out DuckDB?
- Which work stays in pandas?
- What goes into the decision record?
From the lesson pandas, polars or DuckDB: a measured guide
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes

All Data Analysis with Python (pandas, polars, DuckDB) interview questions · The Data Analysis with Python (pandas, polars, DuckDB) track

Warm-up questions

Why is a vectorised pandas expression faster than the same calculation in apply(axis=1)?

You read a large CSV in chunks and compute each chunk's mean order value. Why is the average of those chunk means usually wrong, and how do you get the exact mean?

What is Apache Arrow, and why does it matter when one workflow uses pandas, polars and DuckDB?

Questions with model answers

A pricing step runs df.apply(price_row, axis=1) over 4 million order lines. price_row is an if/elif chain on the customer tier, the quantity and the product category, and the step takes six minutes. How do you rewrite it, and how do you prove that no price changed?

df.memory_usage(deep=True).sum() reports 1.2 GB, but the Python process peaks at 4.5 GB while it loads the CSV and stays at 3 GB afterwards. Explain the gap, and how you bring the peak down.

You need the exact median and 95th percentile of order value per city from a 60 GB CSV on a 16 GB laptop. A colleague's chunked pandas script takes the median of the chunk medians. What is wrong with it, and what do you do instead?

A DuckDB job joins a 300 GB Parquet event table to a 40 GB user table and aggregates by country. On a 64 GB machine shared with other services it fails with an Out of Memory Error, and on some nights the process is simply killed. What do you change, and in what order?

An analyst's notebook, built on a 1 % sample of orders, reports that sales in a small city grew 40 % from one month to the next. On the full data the growth was 6 %. How does that happen, and how do you set up sample-based exploration so that it cannot mislead a report?

A colleague's notebook shows polars running 10 times faster than pandas on "our workload" and proposes moving every report to polars. What do you check before agreeing, and how would you run a fair comparison?

After DuckDB is added to a pandas 3 pipeline, a query on a 1,000,000-row DataFrame takes about 0.2 seconds per call instead of milliseconds, inside a loop that runs 2,000 times. What is happening, and how do you fix it?

Your analytics team of eight writes pandas notebooks, and data volumes have grown tenfold in a year. Propose a default engine and a policy for exceptions, and say how you would know after six months whether the change worked.