Python Module 8 – Files, paths and data formats
Reading and writing text files safely
Open text files in Python with the right mode and encoding, read them whole or line by line, close them with with, and save changes atomically.
What you will learn
- Choose the open() mode that does what you need to a file that exists
- Read and write text with an explicit encoding, and recognise mojibake
- Read whole files, single lines and large files in little memory
- Predict the line endings that text mode reads and writes
- Use with so files close on every path, and save a file atomically
Before you start
What you’ll learn · Before you start
What you will learn
- Choose the open() mode that does what you need to a file that exists
- Read and write text with an explicit encoding, and recognise mojibake
- Read whole files, single lines and large files in little memory
- Predict the line endings that text mode reads and writes
- Use with so files close on every path, and save a file atomically
Before you start
On this page
open("notes.txt", "w") empties a file the moment it opens it, so a program that fails before its first write()
has already lost every line. Reading has traps of its own: the 12 bytes that hold Café ₹250 come back as
Café ₹250 when a program reads them as Windows-1252, the default of many Windows computers, and read() needs
about twice a file’s size in memory.
The safe way to read a text file is open(path, encoding="utf-8") in a with statement, then read() for a small
file or a loop over the file object: What open() returns: an object with methods such as read(), write() and close() for one open file, which keeps its own position in that file. In text mode it reads and writes str; in binary mode, bytes. for one line at a time. To write one, open it with
"w" to replace it or "a" to add to its end. Below you will step through each mode on a real file, watch bytes turn
into the wrong text, measure what reading costs, and save a file so that a crash never leaves half of it behind.
The mode decides what open() does to an existing file
open() takes the file’s path, a mode and, for text, an encoding: A rule for writing text as bytes and reading it back, such as UTF-8, UTF-16 or Latin-1. str.encode() and bytes.decode() take its name, and the bytes themselves never say which encoding wrote them. More in “Text vs bytes: Unicode, encodings, bytearray”, and returns a file
object: the thing you read from and write to. It opens whatever path it is given, so a name that a user typed, such
as ../../secret.txt, can point outside your folder; the next lesson, on pathlib, shows how to refuse one. The mode
is a short string, and each one treats an existing file differently:
| Mode | Opens the file for | If the file exists | If it does not exist |
|---|---|---|---|
"r", the default |
reading | reads it from the start | raises FileNotFoundError |
"w" |
writing | empties it at once | creates it |
"a" |
appending | writes after its last byte | creates it |
"x" |
writing a new file | raises FileExistsError |
creates it |
Two letters combine with these. "b" opens the file in binary mode ("rb", "wb"), where you read and write
bytes instead of str; a later lesson of this module is about binary files. "+" opens it for reading and writing
at once: "r+" keeps the contents, while "w+" empties the file like "w". Step through one diary file opened in
all four modes, and watch what it holds after each call:
The drawing shows the whole scene; its 7 steps are listed below.
Every step as text
- "w" creates the file. open("diary.txt", "w") finds no file, so it creates an empty one and returns a file object to write to.
- Two lines. Two write() calls add Monday and Tuesday. The with block ends and closes the file.
- "a" keeps the lines. Mode "a" opens the file without touching what is in it, and write() adds Wednesday after its last byte.
- "r" reads. With no mode, open() reads. read() returns the three lines as one string, and the program prints them.
- "x" refuses. Mode "x" only creates new files. diary.txt exists, so open() raises FileExistsError before anything is written, and the file keeps its three lines.
- "w" empties at once. Mode "w" again. The file is emptied the moment it is opened, before the first write(), with no error and no warning.
- Only Thursday. write() adds the one new line, and reading the file back shows that it is all the file holds.
Here is the same program, to run and to change:
# "w" creates the file, or empties it if it already exists.
with open("diary.txt", "w", encoding="utf-8") as f:
f.write("Mon: installed Python\n")
f.write("Tue: wrote a loop\n")
# "a" adds to the end and keeps what is there.
with open("diary.txt", "a", encoding="utf-8") as f:
f.write("Wed: opened my first file\n")
# "r" (the default mode) reads.
with open("diary.txt", encoding="utf-8") as f:
print(f.read(), end="")
# "x" creates a new file, and refuses to touch one that exists.
try:
with open("diary.txt", "x", encoding="utf-8") as f:
f.write("this never happens\n")
except FileExistsError as err:
print("x refused:", type(err).__name__, err.filename)
# Opening with "w" again throws the old lines away at once.
with open("diary.txt", "w", encoding="utf-8") as f:
f.write("Thu: lost the first three days\n")
with open("diary.txt", encoding="utf-8") as f:
print(f.read(), end="")Output
Mon: installed Python Tue: wrote a loop Wed: opened my first file x refused: FileExistsError diary.txt Thu: lost the first three days
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 diary.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The second block kept Monday and Tuesday because "a" writes after what is there. The "x" block refused to start,
so the diary was safe from it. The last "w" threw three days away before writing a single character: there was no
error and no warning, because emptying the file is exactly what "w" promises. Press Edit, change that last "w" to
"a" and run it again: the diary keeps all four days. Use "x" when you create files that must never replace an
older one, such as a report named after today’s run, and keep "w" for files you mean to rewrite.
Note
Examples that you run in the browser write their files to an in-memory file system. The files exist only while the page is open and vanish when you reload it; nothing is written to your computer.
To change one line of a file, no mode does the job on its own:
Opening a file with "w" to change one line
open("config.txt", "w") empties the file before your first write(), so writing the changed line leaves that
line alone in the file. Read the old text with "r", build the new text, then write all of it, through a temporary
file if the old version matters (the last section shows how).
Checkpoint
Check: What each mode does to a file
The same bytes are different text under another encoding
A file holds bytes, not characters. Writing a str to a text file encodes it, turning each character into one or
more bytes, and reading decodes the bytes back into characters. Both directions need the same
encoding, and UTF-8: The Unicode encoding that writes each code point as 1 to 4 bytes: 1 for ASCII, 2 for é, 3 for Devanagari and 4 for most emoji. Any ASCII text is also valid UTF-8, byte for byte. More in “Text vs bytes: Unicode, encodings, bytearray”, which can store every character of every
script, is the one to choose for new files. The lesson Text vs bytes shows how UTF-8
uses one to four bytes per character.
text = "Café ₹250\nनमस्ते\n"
with open("menu.txt", "w", encoding="utf-8") as f:
f.write(text)
with open("menu.txt", "rb") as f: # "b": the bytes exactly as they are on disk
raw = f.read()
print(len(text), "characters were saved as", len(raw), "bytes")
# The right encoding gives the text back.
with open("menu.txt", encoding="utf-8") as f:
print(f.read() == text)
# Windows-1252, a common locale encoding on Windows, reads the same bytes as other characters.
with open("menu.txt", encoding="cp1252", errors="replace") as f:
print(f.readline(), end="")
# Without errors="replace", a byte that means nothing in that encoding stops the read.
try:
with open("menu.txt", encoding="cp1252") as f:
f.read()
except UnicodeDecodeError as err:
print(f"{type(err).__name__}: byte {err.object[err.start]:#04x} at position {err.start} has no meaning in cp1252")
# The bytes behind the garbled text: each character, its UTF-8 bytes, and those bytes read as Windows-1252.
for ch in "Cé₹न":
raw = ch.encode("utf-8")
print(ch, raw.hex(" "), raw.decode("cp1252"))Output
17 characters were saved as 32 bytes True Café ₹250 UnicodeDecodeError: byte 0x8d at position 24 has no meaning in cp1252 C 43 C é c3 a9 é ₹ e2 82 b9 ₹ न e0 a4 a8 न
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 encodings.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Seventeen characters became 32 bytes, because é takes two bytes in UTF-8, ₹ takes three, and each of the six
Devanagari code points of नमस्ते takes three. Read back as UTF-8, the text comes out unchanged. Read as Windows-1252,
the bytes on disk are the same, yet é turns into é and ₹ into ₹: this garbled text is known as
mojibake: Garbled text that comes from decoding bytes with the wrong encoding, such as é where é was meant: the two UTF-8 bytes of é read as Windows-1252. No error is raised. More in “Text vs bytes: Unicode, encodings, bytearray”. The last four lines of the output show why, byte by byte, and the table lines
them up. Windows-1252 gives almost every byte a character of its own, so a character that UTF-8 stores in two or
three bytes comes back as two or three characters:
| Character | Its bytes in UTF-8 | The same bytes read as Windows-1252 |
|---|---|---|
C |
43 |
C |
é |
c3 a9 |
é |
₹ |
e2 82 b9 |
₹ |
न, the first code point of नमस्ते |
e0 a4 a8 |
न |
्, its fourth code point |
e0 a5 8d |
an error: byte 8d means nothing |
The Hindi line shows the second way a wrong encoding fails. The strict reader stopped at byte 0x8d, at position 24,
the last byte of the fourth code point, because Windows-1252 gives that byte no meaning. The option
errors="replace" shows such bytes as � instead of stopping, which helps when you only want to look at a damaged
file.
Saving text that was read with errors="replace"
errors="replace" turns every byte it cannot decode into �, and errors="ignore" drops it; neither keeps the
original bytes anywhere. Use them to look at a damaged file, never on text you will save again: the characters
they replaced are lost for good.
That second encoding was not chosen at random. When you leave encoding out, Python 3.14 uses the computer’s
locale encoding: The encoding a computer's settings name for text, which open() uses in Python 3.14 when no encoding is given: UTF-8 on most Linux and macOS systems, often a legacy code page such as Windows-1252 on Windows., which is UTF-8 on most Linux and macOS systems and often a legacy
code page such as Windows-1252 on Windows. The same script can then read the same file correctly on one computer and
garble it on another, which is why every open() in text mode: How open() opens a file unless its mode contains "b": it decodes the bytes it reads into str and encodes the str it writes, with the file's encoding, and translates line endings on the way. in this lesson names its
encoding. To find the calls that do not, start Python with -X warn_default_encoding, and each one reports an
EncodingWarning:
import subprocess
import sys
# A second Python, started with -X warn_default_encoding, reports every open() without an encoding.
program = 'open("notes.txt", "w").close()\nopen("notes.txt", "w", encoding="utf-8").close()'
result = subprocess.run(
[sys.executable, "-X", "warn_default_encoding", "-c", program],
capture_output=True,
text=True,
)
print(result.stderr, end="")
print("warnings:", result.stderr.count("EncodingWarning"))Output
<string>:1: EncodingWarning: 'encoding' argument not specified warnings: 1
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 encoding_warning.py
Version note
Python 3.15 makes UTF-8 the default encoding of open() whatever the locale (PEP 686); Python 3.14 and older use
the locale encoding. Code that passes encoding="utf-8" behaves the same on every version. When you really want
the locale’s encoding, say so with encoding="locale" (Python 3.10 and later), and EncodingWarning stays quiet.
Checkpoint
Check: Encodings and the default one
read() takes the whole file, a loop takes one line at a time
A file object offers several ways to read, and each returns str in text mode:
with open("todo.txt", "w", encoding="utf-8") as f:
f.write("buy milk\ncall the bank\n\nwater the plants\n")
with open("todo.txt", encoding="utf-8") as f:
whole = f.read() # one str with every line
print(repr(whole))
with open("todo.txt", encoding="utf-8") as f:
first = f.readline() # one line, with its "\n"
rest = f.readlines() # a list of the lines that are left
print(repr(first))
print(rest)
# Looping over the file gives one line at a time: the usual way.
with open("todo.txt", encoding="utf-8") as f:
for number, line in enumerate(f, start=1):
task = line.rstrip("\n")
if task: # skip the empty line
print(number, task)
# At the end of the file, read() and readline() return "" instead of raising an error.
with open("todo.txt", encoding="utf-8") as f:
f.read()
print(repr(f.read()), repr(f.readline()))Output
'buy milk\ncall the bank\n\nwater the plants\n' 'buy milk\n' ['call the bank\n', '\n', 'water the plants\n'] 1 buy milk 2 call the bank 4 water the plants '' ''
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 read_styles.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
read() returns the whole remaining text as one string. readline() returns the next line, and readlines()
returns a list of all the lines that are left; both keep the "\n" at the end of each line. Looping over the file
object gives you one line per pass, which is the usual way to read a text file. Remove the line ending with
line.rstrip("\n") rather than strip(): strip() also removes spaces and tabs at the start of the line, which may
be data.
At the end of file (EOF): The point where a stream has nothing left to read. input() raises EOFError there, sys.stdin.readline() returns an empty string, and a for loop over sys.stdin simply stops. More in “Input and output: print(), input() and stdin”, read() and readline() return an empty string instead of
raising an exception, so a blank line inside a file comes back as "\n" and the end as "". Step through a loop
that relies on that difference, over the same four tasks:
Program · readline_loop.py
with open("todo.txt", "w", encoding="utf-8") as f:
f.write("buy milk\ncall the bank\n\nwater the plants\n")
with open("todo.txt", encoding="utf-8") as f:
line = f.readline()
while line:
print(repr(line))
line = f.readline()
print("after the loop:", repr(line))Variables
- changed in this step
Printed so far
Nothing printed yet'buy milk\n' 'call the bank\n' '\n' 'water the plants\n' after the loop: ''
Every step was recorded from a run with Python 3.14.8 on macOS 26 arm64.
The drawing shows the variables when the program ends, and the program's 21 steps are listed below.
Change the program and run it here. Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Every step as text
- Start. No line has run yet, so there are no variables. Python starts with line 1.
- Line 1.
with open("todo.txt", "w", encoding="utf-8") as f:runs. - Line 2.
f.write("buy milk\ncall the bank\n\nwater the plants\n")leavesfas it was. - Line 1.
with open("todo.txt", "w", encoding="utf-8") as f:runs. The first block is over, so the with statement closes the file, and the text that write() left in the buffer reaches todo.txt. - Line 4.
with open("todo.txt", encoding="utf-8") as f:runs. - Line 5.
line = f.readline()setslineto'buy milk\n'. - Line 6.
while line:tests its condition: it is true (lineis'buy milk\n'), so the body runs. - Line 7.
print(repr(line))prints'buy milk\n'. - Line 8.
line = f.readline()setslineto'call the bank\n'(it was'buy milk\n'). - Line 6.
while line:tests its condition: it is true (lineis'call the bank\n'), so the body runs again (pass 2). - Line 7.
print(repr(line))prints'call the bank\n'. - Line 8.
line = f.readline()setslineto'\n'(it was'call the bank\n'). - Line 6.
while line:tests its condition: it is true (lineis'\n'), so the body runs again (pass 3). Key point The blank line comes back as"\n", a string of one character. It is not empty, so the loop goes on. - Line 7.
print(repr(line))prints'\n'. - Line 8.
line = f.readline()setslineto'water the plants\n'(it was'\n'). - Line 6.
while line:tests its condition: it is true (lineis'water the plants\n'), so the body runs again (pass 4). - Line 7.
print(repr(line))prints'water the plants\n'. - Line 8.
line = f.readline()setslineto''(it was'water the plants\n'). - Line 6.
while line:tests its condition: it is false (lineis''), so the loop ends. Key point At the end of the filereadline()returned"", an empty string. An empty string is false, so the loop stops here, and only here. - Line 4.
with open("todo.txt", encoding="utf-8") as f:runs. - Line 9.
print("after the loop:", repr(line))printsafter the loop: ''. That was the last line, so the program ends. Four lines were read, the blank one included. A loop writtenfor line in f:reads the same lines with less code.
The blank line came back as "\n", which is not empty, so the loop went on; only the end of the file gave "", and
while line: stopped there. That difference is what lets a loop such as while line := f.readline(): stop at the
right moment. The file object also remembers how far it has read: the second read() in the last block of
read_styles.py returned nothing because the first one had already moved to the end. To read again from the top,
open the file again or call f.seek(0).
Large files: loop over them, do not load them
read() and readlines() hold the whole file in memory, which is fine for a settings file and a problem for a
log of several gigabytes. A loop holds one line at a time. This example writes a log of 100,000 lines and counts them
both ways, measuring the memory each way needs with the standard library’s tracemalloc:
import os
import tracemalloc
# A log of 100,000 lines, about 3.6 MB.
with open("visits.log", "w", encoding="utf-8") as f:
for i in range(100_000):
f.write(f"{i:06d} GET /products/{i % 250} 200 user{i % 1000}\n")
size = os.path.getsize("visits.log")
print(f"file size: {size / 1e6:.1f} MB")
# 1. read() the whole file into one string.
tracemalloc.start()
with open("visits.log", encoding="utf-8") as f:
text = f.read()
errors_whole = text.count(" 404 ")
lines_whole = text.count("\n")
_, peak_whole = tracemalloc.get_traced_memory()
tracemalloc.stop()
del text
# 2. Loop over the lines: only one line is in memory at a time.
tracemalloc.start()
lines = 0
with open("visits.log", encoding="utf-8") as f:
for line in f:
lines += 1
_, peak_loop = tracemalloc.get_traced_memory()
tracemalloc.stop()
print("lines counted:", lines_whole, lines)
print(f"peak memory with read(): {peak_whole / 1e6:.1f} MB")
print(f"peak memory line by line: {peak_loop / 1e3:.0f} KB")Output
file size: 3.6 MB lines counted: 100000 100000 peak memory with read(): 7.4 MB peak memory line by line: 149 KB
This output changes from run to run: the measured memory differs a little between Python builds and computers
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 big_file.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Both ways counted the same 100,000 lines. read() needed about twice the size of the file at its peak, because the
bytes it read and the string it decoded were in memory at the same moment; a file of 2 GB would need about 4 GB.
The loop stayed at about 150 KB, and that figure does not grow with the file: a ten times larger log needs the same
memory, only more time. Reach for read() when the file is small and you know it, and for the loop in every other
case.
Calling read() on a file of unknown size
f.read() or f.readlines() on a log that may grow to gigabytes can use up the computer’s memory. Loop over the
file object instead: for line in f: keeps one line at a time.
Checkpoint
Check: Reading whole files and lines
Text mode translates line endings, and write() adds none
Lines end with "\n" on Linux and macOS and traditionally with "\r\n" on Windows. Text mode hides the difference:
# A file saved on Windows usually ends each line with "\r\n".
with open("windows.txt", "wb") as f:
f.write(b"first\r\nsecond\r\n")
with open("windows.txt", encoding="utf-8") as f:
print(repr(f.read())) # text mode turns every line ending into "\n"
print("line endings seen:", repr(f.newlines))
with open("windows.txt", encoding="utf-8", newline="") as f:
print(repr(f.read())) # newline="" leaves them as they are
with open("windows.txt", "rb") as f:
print(f.read()) # binary mode never changes a byteOutput
'first\nsecond\n' line endings seen: '\r\n' 'first\r\nsecond\r\n' b'first\r\nsecond\r\n'
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 newlines.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
By default, reading turns "\r\n" and a lone "\r" into "\n", so your code only ever sees "\n"; the file
object’s newlines attribute records which endings it met. This is called
universal newlines: Reading text so that every kind of line ending counts: in text mode "\n", "\r\n" and a lone "\r" all come back as "\n". The file object's newlines attribute records which endings it met.. When writing, text mode does the opposite and turns each
"\n" into the platform’s line ending, which is "\r\n" on Windows. Passing newline="" switches the translation
off in both directions, and the csv module needs exactly that, because it handles line endings itself (see
CSV files with the csv module). Binary mode never changes a byte.
The line endings you write are yours to add. write() writes exactly the string you give it and returns the number
of characters it wrote. It adds no line ending, and neither does writelines(), whose name suggests otherwise;
print() with file= adds one, as on the screen:
with open("shopping.txt", "w", encoding="utf-8") as f:
count = f.write("milk") # no line ending is added
f.write(", bread\n")
print("eggs", "rice", sep=", ", file=f) # print() adds "\n" for you
f.writelines(["tea\n", "sugar\n"]) # a list of strings, written as they are
print("write() returned", count)
with open("shopping.txt", encoding="utf-8") as f:
print(f.read(), end="")Output
write() returned 4 milk, bread eggs, rice tea sugar
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 write_methods.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
"milk" and ", bread\n" ended up on one line because nothing separated them. When you write lines in a loop, put
the "\n" in the string yourself, as in f.write(f"{name},{score}\n"), or use print(..., file=f). Between your
str values and the disk, a text file object has two layers: the text layer, which encodes and translates line
endings, and a output buffer: Memory where written text waits before it is sent on: a line at a time when standard output is a terminal, in larger blocks for a file or a pipe. print(..., flush=True) sends it at once. More in “Input and output: print(), input() and stdin”, which the next section is about.
What a file opened in text mode does when you write and when you read
Text description of the diagram
The diagram is a column of four boxes, read from top to bottom for writing and from bottom to top for reading.
- Your program works with str values.
- The text layer turns each str into bytes with the encoding you chose, and translates "\n" into the platform's line ending. When reading, it decodes bytes back into str and turns every kind of line ending into "\n".
- The buffer keeps bytes in memory and hands them to the operating system when it is full, when you call flush(), or when the file is closed. When reading, it fetches the file in chunks.
- The file on disk holds only bytes. A file opened in binary mode ("rb", "wb") skips the text layer, so you read and write those bytes unchanged.
Expecting write() to end the line
f.write(name) in a loop puts every name on one line. Write f.write(name + "\n"), or print(name, file=f),
which ends the line for you.
Checkpoint
Check: Line endings in and out
Written text waits in a buffer until the file is closed
Writing does not put the bytes on the disk at once. The file object collects them in a
buffer in memory, the lower layer of the diagram above, and passes them to the
operating system when the buffer is full, when you call flush() and when the file is closed. Until then, other
programs, and even another file object in the same program, see an older version of the file. Step through a write
that waits, a close that delivers it, and a with block that fails:
The drawing shows the whole scene; its 8 steps are listed below.
Every step as text
- Open log.txt. open("log.txt", "w") creates log.txt on the disk, empty, and the file object gets a buffer in memory.
- write() fills the buffer. log.write("started\n") returns at once, but the text only reaches the buffer. The operating system has not been given a single byte yet.
- A second reader. A second file object opens log.txt and reads it. It gets an empty string, because the line is still in the first object's buffer.
- close() flushes. log.close() hands the buffer to the operating system, which puts the bytes in the file, and then closes the file.
- Now it is there. Read again, log.txt holds the line. Closing, or flush(), is what makes written text visible to other readers.
- A with block. The with block opens report.txt and writes "total: 42\n", which waits in the buffer like the first line did.
- An exception. The next line raises ValueError. The exception leaves the block, and on its way out the with statement closes the file, which flushes the buffer first.
- Saved and closed. The except clause prints the message, report.closed is True, and the file holds the line written before the error.
Here is the same run as a program:
# Without with: what you write waits in a buffer until the file is closed.
log = open("log.txt", "w", encoding="utf-8")
log.write("started\n")
with open("log.txt", encoding="utf-8") as reader:
print("before close:", repr(reader.read()))
log.close()
with open("log.txt", encoding="utf-8") as reader:
print("after close: ", repr(reader.read()))
# With with: the file is closed when the block ends, even when a line in it fails.
try:
with open("report.txt", "w", encoding="utf-8") as report:
report.write("total: 42\n")
raise ValueError("the next figure was missing")
except ValueError as err:
print("caught:", err)
print("closed:", report.closed)
with open("report.txt", encoding="utf-8") as f:
print("saved:", repr(f.read()))Output
before close: '' after close: 'started\n' caught: the next figure was missing closed: True saved: 'total: 42\n'
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 closing.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The first reader found the file empty although write() had returned: the text was still in the buffer. Closing the
file flushed it. A file you never close may lose what is still in its buffer when the program is killed, and every
open file takes a handle from the operating system, which allows each program only a limited number. The with
statement makes the close automatic: the file is closed when the block ends, by its last line, a return or an
exception, as the second half of the example shows. After the block the object still exists, but reading from it
raises ValueError, because the file is closed.
Key idea
Open files in a with statement: however the block ends, the file is closed, and the text still in its buffer is
written first.
Checkpoint
Check: Buffers, close() and with
An atomic save never leaves half a file
Rewriting a file in place has a weak moment. "w" empties the file first and fills it afterwards, so a crash, a
power cut, a full disk or an exception in the middle leaves only part of the new data, and none of the old. Settings
files, saved games and downloaded data deserve better. The safe pattern writes the new version to a temporary file in
the same folder and then swaps it in with os.replace(). Step through both ways of saving, cut off at the same
moment, and then a save that completes:
The drawing shows the whole scene; its 9 steps are listed below.
Every step as text
- In place: emptied. save_in_place() opens settings.txt with "w", which empties it at once. The three old settings are gone before the first new one is written.
- Two new lines. The settings arrive one at a time. "theme = dark" and "language = hi" are written to the file.
- Cut off. The third setting never arrives, because ConnectionError is raised. The file keeps half of the new settings and none of the old ones, and the program will read it next time.
- Atomic: same start. The example writes the three old settings back and calls save_atomic() with the same settings, which fail at the same moment.
- A file beside it. NamedTemporaryFile(dir=folder) creates an empty temporary file in the same folder as settings.txt.
- Write the copy. The two settings that arrive go into the temporary file. settings.txt is not touched.
- Cut off again. ConnectionError at the same moment. The except clause deletes the temporary file and raises the error again, so the caller still learns that the save failed.
- A complete save. The last call gets all three new settings. They go into a new temporary file, and flush() and os.fsync() push them out to the disk.
- One step. os.replace() puts the temporary file in place of settings.txt in one step. A program that opens settings.txt sees the old version or the new one, never a mixture.
import os
import tempfile
def save_in_place(path, lines):
"""Rewrite the file directly: it is emptied first, then filled line by line."""
with open(path, "w", encoding="utf-8") as f:
for line in lines:
f.write(line + "\n")
def save_atomic(path, lines):
"""Write a temporary file next to the target, then swap it in with one rename."""
folder = os.path.dirname(os.path.abspath(path))
tmp = tempfile.NamedTemporaryFile("w", encoding="utf-8", dir=folder, suffix=".tmp", delete=False)
try:
with tmp:
for line in lines:
tmp.write(line + "\n")
tmp.flush()
os.fsync(tmp.fileno()) # ask the operating system to put the bytes on the disk now
os.replace(tmp.name, path) # the old file is replaced in one step
except BaseException:
os.remove(tmp.name) # never leave the half-written copy behind
raise
def settings():
"""New settings, but the third value fails to arrive, like a crash halfway through a save."""
yield "theme = dark"
yield "language = hi"
raise ConnectionError("the server stopped answering")
for save in (save_in_place, save_atomic):
with open("settings.txt", "w", encoding="utf-8") as f:
f.write("theme = light\nlanguage = en\nfont_size = 14\n")
try:
save("settings.txt", settings())
except ConnectionError as err:
print(f"{save.__name__} failed: {err}")
with open("settings.txt", encoding="utf-8") as f:
print(" the file now holds:", f.read().splitlines())
print(" files in the folder:", sorted(name for name in os.listdir() if not name.endswith(".py")))
# A save that gets every setting: the new file replaces the old one in one step.
save_atomic("settings.txt", ["theme = dark", "language = hi", "font_size = 16"])
with open("settings.txt", encoding="utf-8") as f:
print("save_atomic succeeded:", f.read().splitlines())
print(" files in the folder:", sorted(name for name in os.listdir() if not name.endswith(".py")))Output
save_in_place failed: the server stopped answering the file now holds: ['theme = dark', 'language = hi'] files in the folder: ['settings.txt'] save_atomic failed: the server stopped answering the file now holds: ['theme = light', 'language = en', 'font_size = 14'] files in the folder: ['settings.txt'] save_atomic succeeded: ['theme = dark', 'language = hi', 'font_size = 16'] files in the folder: ['settings.txt']
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 atomic_save.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The in-place save lost the old font size and kept a half-updated file that the program would happily read next time. The atomic save failed at the same moment, yet the file still held the complete old settings, and the temporary file was deleted; the last save then replaced all three settings at once. Four details make it work:
- The temporary file is created in the same folder as the target, so it is on the same file system and the final step is a rename, not a copy.
flush()andos.fsync()push the new bytes out of Python’s buffer and ask the operating system to write them to the disk before the swap.os.replace()puts the new file in place of the old one. The documentation ofos.replace()states that a successful rename is atomic operation: An operation that other programs see either not at all or complete, never half done. On POSIX systems a successful os.replace() is atomic, so a program that opens the file sees the old version or the new one. on POSIX systems such as Linux and macOS: a program that opens the file sees the old version or the new one, never a mixture. It also overwrites an existing file on Windows, whereos.rename()raisesFileExistsErrorinstead.- If anything fails, the
exceptclause removes the temporary file and raises the error again, so the caller still learns that the save failed.
Key idea
To replace a file that matters, write the new version beside it and swap it in with os.replace(): a reader then
finds the old file or the new one, never half of either.
Checkpoint
Check: Saving a file atomically
Practise: tail a log, save atomically
The exercise asks for two functions that use this lesson’s ideas. tail() returns the last lines of a text file
while keeping only those lines in memory, so it works on a log of any size, and atomic_write() replaces a file so
that it never holds part of the old or the new text. Its sample tests run in your browser, one of them on a file of
about 3 MB:
Exercise · Medium · Python
Read the last lines and save a file atomically
Write two functions in textfiles.py. Both read and write text as UTF-8.
tail(path, n)returns a list of the lastnlines of the text file atpath, oldest first, without their line endings. A file with fewer thannlines gives all of them, andnof 0 gives[]. Empty lines count as lines, and a last line without"\n"is still a line. The function must work on a file far larger than the memory you want to spend: keep at mostnlines at a time, never the whole file. One sample test measures this on a file of about 3 MB.atomic_write(path, text)replaces the file atpath(or creates it) withtext, so that the file holds either all of the old text or all of the new text, never part of either. Write a temporary file in the same folder, then move it overpathin one step. If anything fails,pathkeeps its old contents and no temporary file is left in the folder.textmust be astr: anything else, such as123, raisesTypeError, and the tests use that to make a write fail halfway.
Starter code · textfiles.py
def tail(path, n):
"""Return the last n lines of the text file at path, without their line endings."""
# Replace these lines with your code.
with open(path, encoding="utf-8") as f:
return f.read().splitlines()[:n]
def atomic_write(path, text):
"""Replace the file at path with text, so that it never holds part of the old or the new text."""
# Replace these lines with your code.
with open(path, "w", encoding="utf-8") as f:
f.write(text)The sample tests · test_textfiles.py
import os
import tempfile
import tracemalloc
from textfiles import atomic_write, tail
def write_bytes(folder, name, data):
path = os.path.join(folder, name)
with open(path, "wb") as f:
f.write(data)
return path
def test_tail_last_lines():
"""returns the last n lines, oldest first, without line endings"""
with tempfile.TemporaryDirectory() as folder:
path = write_bytes(folder, "log.txt", b"one\ntwo\nthree\nfour\nfive\n")
assert tail(path, 2) == ["four", "five"]
assert tail(path, 1) == ["five"]
def test_tail_short_and_zero():
"""gives every line when the file is shorter than n, and [] for n == 0"""
with tempfile.TemporaryDirectory() as folder:
path = write_bytes(folder, "log.txt", b"one\ntwo\n")
assert tail(path, 5) == ["one", "two"]
assert tail(path, 0) == []
def test_tail_edges():
"""keeps empty lines, a last line without a newline, Windows line endings and UTF-8 text"""
with tempfile.TemporaryDirectory() as folder:
path = write_bytes(folder, "notes.txt", "a\r\n\r\nनमस्ते\r\nlast".encode("utf-8"))
assert tail(path, 3) == ["", "नमस्ते", "last"]
def test_tail_large_file_small_memory():
"""reads a file of about 3 MB while keeping well under 1 MB in memory"""
with tempfile.TemporaryDirectory() as folder:
path = os.path.join(folder, "big.log")
with open(path, "w", encoding="utf-8") as f:
for i in range(100_000):
f.write(f"{i:06d} GET /products/{i % 250} 200\n")
tracemalloc.start()
try:
last = tail(path, 3)
peak = tracemalloc.get_traced_memory()[1]
finally:
tracemalloc.stop()
assert last == ["099997 GET /products/247 200", "099998 GET /products/248 200", "099999 GET /products/249 200"]
assert peak < 1_000_000, f"tail() used {peak:,} bytes at its peak for a {os.path.getsize(path):,}-byte file"
def test_atomic_write_creates_and_replaces():
"""creates the file, replaces it, writes UTF-8 and leaves no other file in the folder"""
with tempfile.TemporaryDirectory() as folder:
path = os.path.join(folder, "settings.txt")
atomic_write(path, "theme = light\n")
atomic_write(path, "theme = dark\nभाषा = hi\n")
with open(path, "rb") as f:
assert f.read() == "theme = dark\nभाषा = hi\n".encode("utf-8")
assert os.listdir(folder) == ["settings.txt"]
def test_failed_write_keeps_the_old_file():
"""a write that fails halfway leaves the old contents and no temporary file"""
with tempfile.TemporaryDirectory() as folder:
path = os.path.join(folder, "settings.txt")
atomic_write(path, "theme = light\n")
try:
atomic_write(path, 123)
except TypeError:
pass
else:
assert False, "atomic_write(path, 123) should raise TypeError"
with open(path, encoding="utf-8") as f:
assert f.read() == "theme = light\n"
assert os.listdir(folder) == ["settings.txt"]A hint
For tail, loop over the open file and append each line to collections.deque(maxlen=n): a full deque drops its oldest item when you append a new one, so it never holds more than n lines. For atomic_write, create the temporary file with tempfile.NamedTemporaryFile("w", encoding="utf-8", dir=folder, delete=False), where folder is the folder of path, write and close it, and only then call os.replace(temporary_name, path). Put the writing in try and delete the temporary file in except before you raise the error again.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
Then test yourself on modes, encodings, reading, line endings, buffers and atomic saves. The practice begins with the idea you know least, asks one question at a time, and explains every choice you make, right or wrong:
Practice that adapts to you
Practise this lesson
18 questions about 6 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
What is the difference between opening a file with “w”, “a” and “x”?
All three open a file for writing and create it when it does not exist; they differ in what they do to a file that
is already there. "w" empties it at once, before anything is written, so the old contents are gone even if you
never call write(). "a" keeps the contents and adds everything you write after the last byte. "x" refuses to
open an existing file and raises FileExistsError, which makes it the safe choice for files that must never replace
an older one. Add "b" to any of them for bytes instead of text, and "+" to read as well; "w+" still empties the
file. To change a file without risking its old contents, write a temporary file and swap it in with os.replace().
Key takeaways
- Open text files with
open(path, mode, encoding="utf-8")in awithstatement:"r"reads,"w"empties and writes,"a"appends, and"x"creates a new file but refuses an existing one. - The same bytes give different text under another encoding; without
encoding=, Python 3.14 uses the locale encoding, which differs between computers, and-X warn_default_encodingfinds the calls that leave it out. read()andreadlines()load the whole file, while a loop over the file object reads one line at a time in a small, fixed amount of memory; a blank line comes back as"\n"and the end of the file as"".- Text mode turns every line ending into
"\n"when reading and back into the platform’s ending when writing,newline=""turns that off, andwrite()adds no line ending of its own. - Written text waits in a buffer until a flush or the close, and a
withblock closes the file however the block ends. - To replace a file safely, write a temporary file in the same folder, call
flush()andos.fsync()on it, and move it into place withos.replace().
Terms in this lesson
- atomic operation
- An operation that other programs see either not at all or complete, never half done. On POSIX systems a successful os.replace() is atomic, so a program that opens the file sees the old version or the new one.
- file object
- What open() returns: an object with methods such as read(), write() and close() for one open file, which keeps its own position in that file. In text mode it reads and writes str; in binary mode, bytes.
- locale encoding
- The encoding a computer's settings name for text, which open() uses in Python 3.14 when no encoding is given: UTF-8 on most Linux and macOS systems, often a legacy code page such as Windows-1252 on Windows.
- text mode
- How open() opens a file unless its mode contains "b": it decodes the bytes it reads into str and encodes the str it writes, with the file's encoding, and translates line endings on the way.
- universal newlines
- Reading text so that every kind of line ending counts: in text mode "\n", "\r\n" and a lone "\r" all come back as "\n". The file object's newlines attribute records which endings it met.
References
- The Python Tutorial: reading and writing files (Python Software Foundation)
- Built-in Functions: open() (Python Software Foundation)
- io: text files, buffering and newline handling (Python Software Foundation)
- io: the opt-in EncodingWarning (Python Software Foundation)
- Glossary: universal newlines (Python Software Foundation)
- os.replace() (Python Software Foundation)
- os.rename() (Python Software Foundation)
- os.fsync() (Python Software Foundation)
- tempfile.NamedTemporaryFile() (Python Software Foundation)
- tracemalloc: trace memory allocations (Python Software Foundation)
- PEP 686: Make UTF-8 mode default (Python Software Foundation)
- PEP 597: Add optional EncodingWarning (Python Software Foundation)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress