Python Module 8 – Files, paths and data formats
Binary files, struct and checksums
Read and write binary files in Python, pack and unpack bytes with struct and int.to_bytes, parse a PNG header, and verify files with SHA-256 checksums.
What you will learn
- Read and write binary data with bytes, int.to_bytes() and struct
- Parse a binary file header, checking its signature, lengths and CRC before its fields
- Verify a file with a SHA-256 checksum and say what a match proves
- Convert binary data to text with base64 and explain why it is not encryption
- Slice a large buffer with memoryview without copying it
Before you start
What you’ll learn · Before you start
What you will learn
- Read and write binary data with bytes, int.to_bytes() and struct
- Parse a binary file header, checking its signature, lengths and CRC before its fields
- Verify a file with a SHA-256 checksum and say what a match proves
- Convert binary data to text with base64 and explain why it is not encryption
- Slice a large buffer with memoryview without copying it
Before you start
On this page
Every PNG image starts with the same 8 bytes, and 8 bytes later come its width and height, two 4-byte numbers written
most significant byte first. Trust those bytes without checking them, and one changed byte makes a program report a width
of 131,392 for an image 320 pixels wide, as this lesson’s PNG example shows. Images, archives, databases and records
written by devices are binary files of this kind, and Python reads them as bytes.
You will read and write bytes, turn numbers and whole records into bytes and back with int.to_bytes() and struct,
parse a PNG header that refuses damaged files, verify a download with a SHA-256 checksum, carry bytes inside text
with base64, and slice large buffers without copying them.
Binary mode reads and writes bytes, untouched
open(path, "rb") and open(path, "wb") open a file in binary mode. read() then returns bytes, the immutable
sequence of numbers from 0 to 255 that the lesson Text vs bytes introduced, and
write() takes bytes; nothing is decoded, encoded or translated on the way, as it would be in
text mode: How open() opens a file unless its mode contains "b": it decodes the bytes it reads into str and encodes the str it writes, with the file's encoding, and translates line endings on the way. More in “Reading and writing text files safely”. Every PNG image starts with the same eight bytes, its
file signature: Fixed bytes at the start of a file that name its format, such as the 8 bytes 89 50 4e 47 0d 0a 1a 0a that begin every PNG image. Checking them first tells a PNG from a JPEG before any field is read., which make a good first file:
# The first eight bytes of every PNG image, written and read back in binary mode.
signature = bytes([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A])
with open("start.bin", "wb") as f: # "b": bytes in, bytes out, nothing translated
print("wrote", f.write(signature), "bytes")
with open("start.bin", "rb") as f:
raw = f.read()
print(raw) # bytes that are printable ASCII are shown as characters
print(raw[0], raw[1:4]) # one byte is an int from 0 to 255; a slice is bytes
print(raw.hex(" "))
print(bytes.fromhex("89 50 4e 47") == raw[:4])
with open("start.bin", "rb") as f:
f.seek(1) # move to byte 1
print(f.read(3), "and now at byte", f.tell())
# Integers need a size and a byte order to become bytes.
n = 1_000_000
print("big-endian: ", n.to_bytes(4, "big").hex(" "))
print("little-endian:", n.to_bytes(4, "little").hex(" "))
print(int.from_bytes(b"\x00\x0f\x42\x40", "big"))
try:
(300).to_bytes(1, "big")
except OverflowError as err:
print("OverflowError:", err)Output
wrote 8 bytes b'\x89PNG\r\n\x1a\n' 137 b'PNG' 89 50 4e 47 0d 0a 1a 0a True b'PNG' and now at byte 4 big-endian: 00 0f 42 40 little-endian: 40 42 0f 00 1000000 OverflowError: int too big to convert
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 bytes_basics.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Python prints a bytes value with the printable ASCII bytes as characters and the rest as escapes such as \x89 and
\r, which is why the signature shows PNG in the middle. The table gives each of the eight bytes three ways:
| Byte | Hex | As an int |
As Python prints it |
|---|---|---|---|
| 0 | 89 |
137 | \x89 |
| 1 | 50 |
80 | P |
| 2 | 4e |
78 | N |
| 3 | 47 |
71 | G |
| 4 | 0d |
13 | \r |
| 5 | 0a |
10 | \n |
| 6 | 1a |
26 | \x1a |
| 7 | 0a |
10 | \n |
Indexing gives one byte as an int, so raw[0] is 137, and slicing gives bytes, so raw[1:4] is b'PNG'.
hex() and bytes.fromhex() convert to and from the hexadecimal text that file-format specifications and hex
viewers use. seek(offset) moves the file’s position and tell() reports it, so a program can jump to the part of a
file it needs and read(n) exactly n bytes from there.
Mixing text and bytes
b"PNG" + "!" raises TypeError: bytes and text never mix. Encode the text or decode the bytes first, and open
binary files with "rb" or "wb", never in text mode, where decoding fails or garbles the data and newline
translation changes bytes.
Checkpoint
Check: Bytes in binary mode
An integer becomes bytes only with a size and an order
The last lines of that example turned one million into four bytes twice. An integer has no single binary form: you choose how many bytes it takes and in which byte order: The order in which the bytes of a number wider than one byte are stored. Big-endian puts the most significant byte first and little-endian the least significant, so one million is 00 0f 42 40 in one and 40 42 0f 00 in the other. they go. Big-endian puts the most significant byte first, as people write numbers; little-endian puts it last, as the processors of most computers and phones store numbers in memory. A file format always says which one it uses. Step through what happens when bytes are read back in the other order:
Program · byte_order.py
n = 1_000_000
big = n.to_bytes(4, "big")
little = n.to_bytes(4, "little")
print("big: ", big.hex(" "))
print("little:", little.hex(" "))
back = int.from_bytes(big, "big")
bad = int.from_bytes(little, "big")
print(back, bad)Variables
- changed in this step
Printed so far
Nothing printed yetbig: 00 0f 42 40 little: 40 42 0f 00 1000000 1078071040
Every step was recorded from a run with Python 3.14.8 on macOS 26 arm64.
The drawing shows the variables when the program ends, and the program's 9 steps are listed below.
Change the program and run it here. Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Every step as text
- Start. No line has run yet, so there are no variables. Python starts with line 1.
- Line 1.
n = 1_000_000setsnto1000000. - Line 2.
big = n.to_bytes(4, "big")setsbigtob'\x00\x0fB@'. Big-endian puts the most significant byte first, as people write numbers, 00 0f 42 40. Python shows the bytes 0x42 and 0x40 as the characters B and @. - Line 3.
little = n.to_bytes(4, "little")setslittletob'@B\x0f\x00'. Little-endian stores the same four bytes the other way round, 40 42 0f 00, the order most computers and phones use for numbers in memory. - Line 4.
print("big: ", big.hex(" "))printsbig: 00 0f 42 40. - Line 5.
print("little:", little.hex(" "))printslittle: 40 42 0f 00. - Line 6.
back = int.from_bytes(big, "big")setsbackto1000000. Key point Read in the order they were written in, the four bytes give back exactly 1,000,000. - Line 7.
bad = int.from_bytes(little, "big")setsbadto1078071040. The bug The little-endian bytes read as big-endian give 1,078,071,040, and nothing fails. The bytes themselves never say which order wrote them. - Line 8.
print(back, bad)prints1000000 1078071040. That was the last line, so the program ends. A file format states its byte order. Read and write numbers in that order, whatever order the computer uses.
Read in the order they were written in, the bytes gave back 1,000,000; read in the other order, the same four bytes
gave 1,078,071,040, and nothing failed. Press Edit, change "big" to "little" on the line that sets bad, and run
it: the second number is 1,000,000 too. n.to_bytes(length, order) and int.from_bytes(data, order) take the order
as an argument, a number that does not fit the size you give raises OverflowError, as the first example showed,
and signed=True allows negative numbers.
Checkpoint
Check: Byte order of integers
struct packs a whole record from one format string
A record of several fields, such as a sensor reading, would need one to_bytes() call per field. The struct
module packs and unpacks the whole record from a format string:
import struct
# One reading: sensor id (unsigned 16-bit), seconds since start (unsigned 32-bit), temperature (32-bit float).
RECORD = struct.Struct("<HIf") # "<": little-endian, standard sizes, no padding
print("record size:", RECORD.size, "bytes")
readings = [(7, 0, 21.5), (7, 60, 22.7), (12, 60, -3.25)]
with open("readings.bin", "wb") as f:
for reading in readings:
f.write(RECORD.pack(*reading))
with open("readings.bin", "rb") as f:
data = f.read()
print("file size:", len(data), "bytes")
print("first record:", data[: RECORD.size].hex(" "))
for sensor, seconds, temperature in RECORD.iter_unpack(data):
print(f"sensor {sensor:>2} at {seconds:>2} s: {temperature}")
# The same numbers in big-endian ("network") order, and with native alignment.
print(struct.pack(">H", 7).hex(), struct.pack("<H", 7).hex())
print("calcsize: '<BI' =", struct.calcsize("<BI"), "| '@BI' =", struct.calcsize("@BI"))
try:
RECORD.unpack(data[:8]) # a truncated record
except struct.error as err:
print("struct.error:", err)Output
record size: 10 bytes file size: 30 bytes first record: 07 00 00 00 00 00 00 00 ac 41 sensor 7 at 0 s: 21.5 sensor 7 at 60 s: 22.700000762939453 sensor 12 at 60 s: -3.25 0007 0700 calcsize: '<BI' = 5 | '@BI' = 8 struct.error: unpack requires a buffer of 10 bytes
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 struct_records.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The format "<HIf" reads as: little-endian with standard sizes, then an unsigned 16-bit integer, an unsigned 32-bit
integer and a 32-bit float, 10 bytes in all. The first record of the file shows each field in its place, and each
number little-endian, so the 7 comes first in 07 00:
| Bytes | Field | Letter | In the first record | Value |
|---|---|---|---|---|
| 0 to 1 | sensor id | H |
07 00 |
7 |
| 2 to 5 | seconds since start | I |
00 00 00 00 |
0 |
| 6 to 9 | temperature | f |
00 00 ac 41 |
21.5 |
The first character of a format sets the byte order and whether the platform’s sizes and alignment apply:
| First character | Byte order | Sizes and padding |
|---|---|---|
< |
little-endian | standard sizes, no padding |
> or ! |
big-endian (! is called network order) |
standard sizes, no padding |
= |
the computer’s own | standard sizes, no padding |
@, or nothing |
the computer’s own | the C compiler’s sizes, with padding for alignment |
The common letters are B, H, I and Q for unsigned integers of 1, 2, 4 and 8 bytes, the lower-case b, h,
i and q for signed ones, f and d for 4- and 8-byte floats, ? for a bool, s for a run of bytes ("4s"
is four bytes) and x for a padding byte. The last lines of the output show why files and network protocols must
start their format with <, > or !: with the default @, "BI" takes 8 bytes instead of 5, because the
integer is moved to a position divisible by 4, and the sizes depend on the computer.
Three more details came out of the run. A 32-bit float holds about seven significant digits, so 22.7 came back as
22.700000762939453; use d when a value must survive exactly as a Python float. Creating a struct.Struct once
compiles its format for every later pack() and unpack(), and iter_unpack() walks a buffer record by record.
And unpack() raises struct.error unless the buffer has exactly the record’s size, while
unpack_from(buffer, offset) reads one record at any offset of a larger buffer.
Leaving the byte order to the computer
struct.pack("BI", 1, 7) packs 8 bytes in this computer’s own order and layout, which another computer may read
differently. In files and network formats, start every format with <, > or !, as the format defines.
Checkpoint
Check: Records with struct
A header parser checks before it trusts a field
Most binary formats start with a header: a fixed signature that says what kind of file it is, then fields at fixed offsets. The PNG specification of the W3C defines the start of every PNG file. After the signature comes the IHDR chunk: 4 bytes of length, 4 bytes of chunk type, 13 bytes of data that start with the width and the height, and a 4-byte CRC: A cyclic redundancy check: a short checksum, such as the 32-bit value of zlib.crc32(), that detects accidental changes to data. Anyone can compute the CRC of changed data, so it does not detect a deliberate change., a checksum of the chunk’s type and data. Every multi-byte number is big-endian:
Where the width and height sit in a PNG file
Text description of the diagram
The diagram is a column of seven fields, in the order they appear in the file.
- Bytes 0 to 7: the PNG signature, 89 50 4e 47 0d 0a 1a 0a.
- Bytes 8 to 11: the length of the first chunk's data, 13, as a big-endian unsigned integer.
- Bytes 12 to 15: the chunk type, the ASCII letters IHDR.
- Bytes 16 to 19: the image width, a big-endian unsigned integer.
- Bytes 20 to 23: the image height, a big-endian unsigned integer.
- Bytes 24 to 28: five one-byte settings: bit depth, colour type, compression method, filter method and interlace method.
- Bytes 29 to 32: the CRC-32 checksum of the chunk type and data, bytes 12 to 28.
This program builds a small but complete PNG image in memory, writes it, reads back only the first 33 bytes and finds
the image’s size in them. Then it gives three files that are not good PNGs to two parsers: trusting_size(), which
reads bytes 16 to 23 and checks nothing, and png_size(), which checks first:
import struct
import zlib
PNG_SIGNATURE = b"\x89PNG\r\n\x1a\n"
def chunk(kind, payload):
"""A PNG chunk: length (4 bytes), type (4), data, CRC-32 of type and data (4), all big-endian."""
return struct.pack(">I", len(payload)) + kind + payload + struct.pack(">I", zlib.crc32(kind + payload))
def make_png(width, height):
"""A valid grey image, built in memory."""
ihdr = struct.pack(">IIBBBBB", width, height, 8, 0, 0, 0, 0) # 8-bit greyscale, no interlacing
rows = b"".join(b"\x00" + bytes([128]) * width for _ in range(height)) # filter byte + pixels
return PNG_SIGNATURE + chunk(b"IHDR", ihdr) + chunk(b"IDAT", zlib.compress(rows)) + chunk(b"IEND", b"")
def png_size(data):
"""Width and height from the IHDR chunk, which must come right after the signature."""
if not data.startswith(PNG_SIGNATURE):
raise ValueError("not a PNG file")
if len(data) < 33:
raise ValueError("too short for a PNG header")
length, kind = struct.unpack_from(">I4s", data, 8)
if kind != b"IHDR" or length != 13:
raise ValueError("the first chunk is not a valid IHDR")
(crc,) = struct.unpack_from(">I", data, 29)
if crc != zlib.crc32(data[12:29]):
raise ValueError("the IHDR chunk is damaged (CRC mismatch)")
width, height = struct.unpack_from(">II", data, 16)
return width, height
def trusting_size(data):
"""What a parser that trusts the file does: read bytes 16 to 23 as two numbers and check nothing."""
return struct.unpack(">II", data[16:24])
image = make_png(320, 200)
with open("grey.png", "wb") as f:
f.write(image)
with open("grey.png", "rb") as f:
head = f.read(33) # only the header is needed, not the whole image
print("file size:", len(image), "bytes; header:", head[:24].hex(" "))
print("size:", png_size(head))
for name, data in [("a JPEG", b"\xff\xd8\xff\xe0" + bytes(40)), ("cut off", image[:20]),
("one byte changed", image[:17] + b"\x02" + image[18:])]:
try:
trusting = trusting_size(data)
except struct.error as err:
trusting = f"struct.error: {err}"
try:
checked = png_size(data)
except ValueError as err:
checked = f"ValueError: {err}"
print(f"{name}:\n trusting: {trusting}\n checked: {checked}")Output
file size: 657 bytes; header: 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 01 40 00 00 00 c8 size: (320, 200) a JPEG: trusting: (0, 0) checked: ValueError: not a PNG file cut off: trusting: struct.error: unpack requires a buffer of 8 bytes checked: ValueError: too short for a PNG header one byte changed: trusting: (131392, 200) checked: ValueError: the IHDR chunk is damaged (CRC mismatch)
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 png_size.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The trusting parser reported a size of (0, 0) for a JPEG and a width of 131,392 for the image with one changed byte,
and crashed with struct.error on the cut-off file. png_size() refused all three, each with a reason. Step through
its checks on the same four inputs:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- A JPEG. png_size() first compares the first 8 bytes with the PNG signature. A JPEG starts ff d8 ff e0, so it is refused at once. The trusting parser read bytes 16 to 23 of it anyway and reported (0, 0).
- Cut off. The first 20 bytes of the image start with the signature but stop inside the width. The length check refuses them before unpack_from() runs out of bytes; the trusting parser stopped with struct.error.
- One byte changed. Byte 17 changed from 00 to 02. The signature, the length and the chunk type are fine, but the CRC of bytes 12 to 28 no longer matches, so the width is never read. The trusting parser reported a width of 131,392.
- The real image. The first 33 bytes of the 320 × 200 image pass the signature and length checks, and bytes 8 to 15 give the length 13 and the type IHDR, so bytes 16 to 23 really are a width and a height.
- The CRC matches. zlib.crc32() of bytes 12 to 28 equals the CRC stored at bytes 29 to 32, so no byte of the chunk's type and data has changed.
- Only now the size. unpack_from('>II', data, 16) reads 00 00 01 40 and 00 00 00 c8, two big-endian numbers: the width 320 and the height 200.
The signature tells a PNG from a JPEG; the length check stops a cut-off file before unpack_from() runs out of
bytes; the chunk type and length make sure the bytes at offset 16 really are a width; and the CRC, computed with
zlib.crc32() exactly as the specification describes, catches the file in which a single byte of the width had
changed. The same habit protects any parser of files from outside: check lengths before slicing, check signatures
and checksums before using fields, and put limits on sizes that you are about to allocate memory for.
Key idea
Every length, offset and size in a file was chosen by whoever made the file: check the signature, the lengths and the checksum before you use a single field.
Checkpoint
Check: Parsing a header safely
A SHA-256 digest changes when any byte changes
A cryptographic hash function: A function such as SHA-256 that turns data of any size into a short digest, 32 bytes for SHA-256, so that any change to the data changes the digest and two inputs with the same digest cannot feasibly be found. turns data of any size into a short digest, 32 bytes for SHA-256, written as 64 hexadecimal characters. Changing any bit of the input gives a completely different digest, and finding two inputs with the same SHA-256 digest is not feasible, so publishing the digest of a file lets anyone check their copy:
import hashlib
import hmac
with open("release.zip", "wb") as f: # stands in for a download: 3 MB of known bytes
for i in range(3 * 1024):
f.write(bytes([i % 256]) * 1024)
with open("release.zip", "rb") as f:
digest = hashlib.file_digest(f, "sha256") # Python 3.11+: reads the file in chunks for you
print("sha256:", digest.hexdigest())
# The same digest by hand, one 64 KB chunk at a time, for code that also reports progress.
sha = hashlib.sha256()
with open("release.zip", "rb") as f:
while chunk := f.read(65536):
sha.update(chunk)
print("same:", sha.hexdigest() == digest.hexdigest())
published = "903743b787fe087d3154c86ccdb37e49bfbf583f68d103467d0e9de5820c907c" # from the download page
print("matches the published value:", hmac.compare_digest(digest.hexdigest(), published))
# One changed byte gives a completely different digest.
with open("release.zip", "r+b") as f:
f.seek(1_000_000)
f.write(b"\xff")
with open("release.zip", "rb") as f:
damaged = hashlib.file_digest(f, "sha256").hexdigest()
print("after one byte changed:", damaged)
print("matches the published value:", hmac.compare_digest(damaged, published))Output
sha256: 903743b787fe087d3154c86ccdb37e49bfbf583f68d103467d0e9de5820c907c same: True matches the published value: True after one byte changed: 701eb10797c106c7486090e45374d1b83270f4c05fcd9cb7107efb5387c160b3 matches the published value: False
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 checksum.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
hashlib.file_digest(), added in Python 3.11, reads the file in chunks, so a file of several gigabytes needs no more
memory than a small one. The loop with update() gives the same digest and is the shape to use when you also want to
report progress. Compare digests with hmac.compare_digest(): it is designed to take the same time whether the first
or the last character differs, which matters when the value is a secret token or a signature that an attacker could
guess byte by byte, and costs nothing for a public checksum. Press Edit and write b"\xd0" instead of b"\xff" at
offset 1,000,000: that byte already holds 0xd0, so the file stays the same and the digest matches again.
What a checksum proves depends on where its published value came from. Step through a download whose digest comes from the project’s own site and whose file comes from a mirror:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- The published digest. The project publishes the SHA-256 digest of release.zip on its HTTPS site, 64 hexadecimal characters that start 9037 and end 907c.
- The download. The 3 MB file itself comes from a mirror, which may be any server on the way.
- Hashing in chunks. hashlib.file_digest() reads the file in chunks, so memory stays at about one chunk however large the file is. The digest it gives starts 9037 and ends 907c.
- Equal digests. hmac.compare_digest() finds the two digests equal, so this file is, byte for byte, the file whose digest the project published.
- One byte changed. This time one byte at offset 1,000,000 changed on the way. The digest becomes 701e…60b3, nothing like the published one, so the program refuses the file.
- Where the digest came from. Had the mirror also served the digest, whoever replaced the file could have replaced the digest too. A digest detects tampering only when it comes from a source you trust; proving who made a file takes a signature.
A checksum always detects a damaged download. It also detects a file that someone changed on the way, but only if the
published value came from a source you trust, such as the project’s HTTPS site, because whoever replaced the file
could replace a checksum stored next to it. Proving who made a file needs a digital signature. And choose SHA-256 or
another SHA-2 or SHA-3 function: the hashlib documentation warns that MD5 and SHA-1 have known collision
weaknesses.
Checkpoint
Check: Checksums with SHA-256
base64 writes bytes as text and hides nothing
JSON, email and URLs carry text, so binary data must be written as text to travel inside them. base64: An encoding that writes every 3 bytes as 4 characters from a set of 64 letters, digits and symbols, so that binary data can travel inside text such as JSON. Anyone can decode it, so it hides nothing., defined in RFC 4648, writes every 3 bytes as 4 characters from a set of 64 letters, digits and symbols:
import base64
import json
thumbnail = bytes(range(256))[::37] # a few bytes that are not text
encoded = base64.b64encode(thumbnail).decode("ascii") # bytes -> base64 text
print(len(thumbnail), "bytes became", len(encoded), "characters:", encoded)
message = json.dumps({"name": "dot.png", "data": encoded}) # now it fits into JSON
received = json.loads(message)
print("round trip ok:", base64.b64decode(received["data"]) == thumbnail)
print("URL-safe:", base64.urlsafe_b64encode(b"\xfb\xff\xfe").decode(), "instead of", base64.b64encode(b"\xfb\xff\xfe").decode())
print("anyone can decode it:", base64.b64decode("c2VjcmV0LXBhc3N3b3Jk"))Output
7 bytes became 12 characters: ACVKb5S53g== round trip ok: True URL-safe: -__- instead of +//+ anyone can decode it: b'secret-password'
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 base64_text.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The table follows the example’s three bytes fb ff fe through the encoding. In a row, their 24 bits are
11111011 11111111 11111110; base64 cuts them into four groups of 6 bits, each a number from 0 to 63 that picks one
character of the alphabet:
| Step | Group 1 | Group 2 | Group 3 | Group 4 |
|---|---|---|---|---|
| 6 bits | 111110 |
111111 |
111111 |
111110 |
| Number | 62 | 63 | 63 | 62 |
| Standard alphabet | + |
/ |
/ |
+ |
| URL-safe alphabet | - |
_ |
_ |
- |
Four characters for every three bytes make the text about a third longer than the bytes, plus = padding when the
last group is short: the 7 bytes of the thumbnail became 12 characters. urlsafe_b64encode() uses - and _
instead of + and /, which have meanings in URLs and file names. And base64 is an encoding, not encryption: the
last line decodes a “hidden” password without any key. Never use it to protect anything.
Checkpoint
Check: base64 is not encryption
A memoryview slices a buffer without copying it
Slicing bytes or a bytearray copies the slice, which costs time and memory when the buffer is large. A
memoryview is a window onto the original buffer, and slicing a view copies nothing:
data = bytearray(b"HEADER" + bytes(10) + b"PAYLOAD")
view = memoryview(data) # a window onto data: slicing it copies nothing
payload = view[16:]
print(payload.nbytes, "bytes in the view:", payload.tobytes())
copy = bytes(data[16:]) # slicing bytes or bytearray makes a copy
data[16:23] = b"payload" # change the buffer itself
print("view sees the change:", payload.tobytes())
print("the copy does not: ", copy)
view[0:6] = b"header" # a view of a bytearray can also write into it
print(data[:6])
try:
data.extend(b"!") # a buffer with views on it cannot change size
except BufferError as err:
print("BufferError:", err)
payload.release()
view.release() # once every view is released, it can
data.extend(b"!")
print(len(data), "bytes now")Output
7 bytes in the view: b'PAYLOAD' view sees the change: b'payload' the copy does not: b'PAYLOAD' bytearray(b'header') BufferError: Existing exports of data: object cannot be re-sized 24 bytes now
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 views.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Step through the same program and watch what each name holds when the buffer changes:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- A view. memoryview(data) is a window onto the bytearray, and slicing the view copies nothing, so payload shows bytes 16 to 22 of data, PAYLOAD.
- A copy. Slicing the bytearray itself makes a copy, and bytes() of it is a new object, so copy holds its own 7 bytes, PAYLOAD.
- data changes. data[16:23] = b'payload' writes into the buffer. payload shows the new bytes at once, because it looks at the same buffer, while copy still holds PAYLOAD.
- Writing through a view. view[0:6] = b'header' writes into data through the view. A view of a bytearray can change it, which is how network and file code fills a large buffer piece by piece.
- No resizing while viewed. data.extend(b'!') raises BufferError while any view exists, because the buffer cannot change size under the views that point into it.
- Release the views. payload.release() and view.release() end both views. Now extend() works, and data has 24 bytes.
The view showed the change made to the bytearray afterwards, while the copy kept the old bytes. A view of a
bytearray can also write into it, which is how network and file code fills a large buffer piece by piece. While any
view exists, the buffer cannot change its size, and BufferError says so; call release() on every view, or use a
view in a with statement, when you are done with it.
Checkpoint
Check: Views without copies
Practise: read a PNG header and verify a checksum
The exercise asks for png_size(), which refuses data that is not a PNG, is cut off, has another first chunk, a CRC
that does not match or a size of 0, and for verify_checksum(), which hashes a file in chunks and compares the digest
with a published value written in any case and with spaces around it. The starter code is this lesson’s trusting
parser. Its sample tests run in your browser:
Exercise · Medium · Python
Read a PNG header and verify a checksum
Write two functions in binfiles.py.
png_size(data)takes the first bytes of a file and returns(width, height)from the PNG header: the 8-byte signature89 50 4e 47 0d 0a 1a 0a, then the IHDR chunk, whose 4-byte length is 13, whose 4-byte type isb"IHDR", whose data starts with the width and the height as big-endian unsigned 32-bit integers, and whose 4-byte CRC, at bytes 29 to 32, iszlib.crc32()of bytes 12 to 28. RaiseValueErrorwhen the data does not start with the signature, is shorter than the 33 bytes this header needs, has another first chunk, has a CRC that does not match, or has a width or height of 0.verify_checksum(path, expected_hex)returnsTruewhen the SHA-256 digest of the file atpathequalsexpected_hex, andFalseotherwise. Accept the expected value in upper or lower case and with spaces or a line break around it, as it often comes from a web page or a.sha256file. Read the file in chunks (hashlib.file_digest()does this), and compare withhmac.compare_digest().
Starter code · binfiles.py
import hashlib
import struct
def png_size(data):
"""(width, height) from a PNG header, or ValueError."""
# Replace these lines with your code.
return struct.unpack(">II", data[16:24])
def verify_checksum(path, expected_hex):
"""Whether the SHA-256 digest of the file at path is expected_hex."""
# Replace these lines with your code.
with open(path, "rb") as f:
return hashlib.sha256(f.read()).hexdigest() == expected_hexThe sample tests · test_binfiles.py
import hashlib
import os
import struct
import tempfile
import zlib
from binfiles import png_size, verify_checksum
SIGNATURE = b"\x89PNG\r\n\x1a\n"
def header(width, height, kind=b"IHDR", length=13, crc=None):
"""The first 33 bytes of a PNG file, with a correct CRC unless one is given."""
body = kind + struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0)
stored = zlib.crc32(body) if crc is None else crc
return SIGNATURE + struct.pack(">I", length) + body + struct.pack(">I", stored)
def raises(data):
try:
png_size(data)
except ValueError:
return True
return False
def test_reads_width_and_height():
"""returns (width, height) from a correct header, with or without more data after it"""
assert png_size(header(320, 200)) == (320, 200)
assert png_size(header(1, 70_000) + b"IDAT and more") == (1, 70_000)
def test_refuses_other_files():
"""raises ValueError for data that does not start with the PNG signature"""
assert raises(b"\xff\xd8\xff\xe0" + bytes(40))
assert raises(b"GIF89a" + bytes(40))
assert raises(b"")
def test_refuses_truncated_and_damaged_headers():
"""raises ValueError for cut-off data, another first chunk, a wrong length, a bad CRC or a zero size"""
good = header(320, 200)
assert raises(good[:20])
assert raises(good[:32])
assert raises(header(320, 200, kind=b"IDAT"))
assert raises(header(320, 200, length=12))
assert raises(header(320, 200, crc=0))
assert raises(header(0, 200))
assert raises(header(320, 0))
def test_checksum():
"""compares the file's SHA-256 with the expected value, ignoring case and surrounding spaces"""
with tempfile.TemporaryDirectory() as folder:
path = os.path.join(folder, "release.zip")
with open(path, "wb") as f:
f.write(bytes(range(256)) * 4096)
expected = hashlib.sha256(bytes(range(256)) * 4096).hexdigest()
assert verify_checksum(path, expected) is True
assert verify_checksum(path, " " + expected.upper() + "\n") is True
assert verify_checksum(path, expected[:-1] + ("0" if expected[-1] != "0" else "1")) is False
assert verify_checksum(path, "") is FalseA hint
Check the length and the signature before you unpack anything: struct.unpack_from(">I4s", data, 8) gives the chunk's length and type, struct.unpack_from(">II", data, 16) the width and height, and struct.unpack_from(">I", data, 29) the stored CRC, which must equal zlib.crc32(data[12:29]). For the checksum, normalise the expected value with expected_hex.strip().lower() before comparing it with digest.hexdigest().
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
Seven ideas, from bytes and byte order to views, are in the practice below. It asks about the one you know least first, moves to harder questions as your answers come out right, and explains every choice as soon as you make it:
Practice that adapts to you
Practise this lesson
15 questions about 7 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
What do the characters <, > and ! mean at the start of a struct format string, and why do they matter?
They set the byte order and switch off the platform’s own layout. < means little-endian and > big-endian; !
is network order, which is big-endian too. With any of them, and with =, every type has its standard size (H 2
bytes, I 4, Q 8) and no padding is added between fields. Without one of these characters, struct uses @: the
computer’s byte order, its C compiler’s sizes and padding for alignment, so "BI" takes 8 bytes instead of 5 and a
long takes 4 or 8 bytes depending on the system. That suits data exchanged with C code on the same computer. For
files and network protocols, which other computers read, always state the order the format defines.
Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.
-
Fresher to mid
An installer downloads a 2 GB file, and the project's website publishes its SHA-256 checksum. How do you verify the file in Python without loading 2 GB into memory, and what does a matching checksum prove and not prove?
-
Mid to senior
You parse an upload format in which each record starts with a 4-byte length followed by that many bytes of data. What can a malformed or malicious file do to a naive parser, and how do you write the parser defensively?
Key takeaways
- Open binary files with
"rb"and"wb":read()givesbytes,write()takesbytes, and nothing is decoded or translated. - An integer becomes bytes only with a size and a byte order, and bytes read in the wrong order give a wrong number without any error.
structpacks and unpacks records from a format string; start it with<,>or!for anything other computers read.- Parse headers defensively: the signature, the lengths and the checksum first, then the fields, with limits on every size you allocate.
hashlib.file_digest(f, "sha256")hashes a file of any size in chunks, and a matching digest proves the file is the published one only when the digest came from a source you trust.- base64 writes every 3 bytes as 4 characters and hides nothing; a
memoryviewslices a large buffer without copying it.
Terms in this lesson
- base64
- An encoding that writes every 3 bytes as 4 characters from a set of 64 letters, digits and symbols, so that binary data can travel inside text such as JSON. Anyone can decode it, so it hides nothing.
- byte order
- The order in which the bytes of a number wider than one byte are stored. Big-endian puts the most significant byte first and little-endian the least significant, so one million is 00 0f 42 40 in one and 40 42 0f 00 in the other.
- CRC
- A cyclic redundancy check: a short checksum, such as the 32-bit value of zlib.crc32(), that detects accidental changes to data. Anyone can compute the CRC of changed data, so it does not detect a deliberate change.
- cryptographic hash function
- A function such as SHA-256 that turns data of any size into a short digest, 32 bytes for SHA-256, so that any change to the data changes the digest and two inputs with the same digest cannot feasibly be found.
- file signature
- Fixed bytes at the start of a file that name its format, such as the 8 bytes 89 50 4e 47 0d 0a 1a 0a that begin every PNG image. Checking them first tells a PNG from a JPEG before any field is read.
References
- struct: interpret bytes as packed binary data (Python Software Foundation)
- int.to_bytes() and int.from_bytes() (Python Software Foundation)
- Binary sequence types: bytes, bytearray, memoryview (Python Software Foundation)
- hashlib.file_digest() (Python Software Foundation)
- hmac.compare_digest() (Python Software Foundation)
- zlib.crc32() (Python Software Foundation)
- base64: Base16, Base32, Base64 and Base85 data encodings (Python Software Foundation)
- Portable Network Graphics (PNG) Specification (Third Edition) (World Wide Web Consortium)
- RFC 4648: The Base16, Base32, and Base64 Data Encodings (Internet Engineering Task Force)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress