Python Module 8 – Files, paths and data formats
pickle, shelve and safe serialisation
Save Python objects with pickle and shelve, see why unpickling untrusted data can run code, sign pickles with HMAC, and know when JSON or SQLite fits better.
What you will learn
- Save and load objects with pickle, and predict what comes back and which Pythons can read the file
- Explain how loading a pickle can import and call any function, and limit it with find_class()
- Protect pickles with an HMAC tag so that changed data is refused before it is unpickled
- Store records in a shelf without losing changes
- Choose JSON, CSV, SQLite or pickle for a given job
Before you start
What you’ll learn · Before you start
What you will learn
- Save and load objects with pickle, and predict what comes back and which Pythons can read the file
- Explain how loading a pickle can import and call any function, and limit it with find_class()
- Protect pickles with an HMAC tag so that changed data is refused before it is unpickled
- Store records in a shelf without losing changes
- Choose JSON, CSV, SQLite or pickle for a given job
Before you start
On this page
One call, pickle.dumps(state), turns a whole game, with its players, a board of tuples and a set of items, into 224
bytes, and pickle.loads() builds equal objects of the same classes from those bytes. The documentation is just as
plain about the danger: the pickle module is not secure, because loading a pickle can run code chosen by whoever made
the bytes, so pickle is only for data your own program wrote and protected.
You will save and restore that game with pickle.dump() and pickle.load(), step through the instructions inside a
pickle to see how loading can call functions, protect pickles with an HMAC tag so that changed bytes are refused, keep
records in shelve, a dictionary on disk whose values are pickles, without losing changes, and choose between
pickle, JSON, CSV and SQLite.
A round trip gives back new, equal objects
The game in this example lives in a module of its own, game.py, and the program saves a whole game state to a file
opened in binary mode, because a pickle is bytes, and reads it back:
game/game_state.py
import pickle
import threading
from game import GameState, Player
asha = Player("Asha", 4, {"lantern", "map"})
ravi = Player("Ravi", 2)
state = GameState(turn=17, players=[asha, ravi], scores={"Asha": 1250, "Ravi": 980},
board=((0, 1, 0), (2, 0, 1)))
state.leader = asha # the same Player object, reachable twice
with open("save.pkl", "wb") as f: # pickles are bytes: open the file in binary mode
pickle.dump(state, f)
with open("save.pkl", "rb") as f:
loaded = pickle.load(f)
print("protocol:", pickle.DEFAULT_PROTOCOL, "| size:", len(pickle.dumps(state)), "bytes")
print("equal:", loaded == state, "| a new object:", loaded is not state)
print("types kept:", type(loaded.players[0]).__name__, type(loaded.board).__name__, sorted(loaded.players[0].items))
print("shared object kept:", loaded.leader is loaded.players[0])
print("class saved by name:", b"game" in pickle.dumps(state) and b"GameState" in pickle.dumps(state))
# Not everything can be pickled: functions are saved by name, and a lambda has none.
state.on_win = lambda player: print(player.name, "wins")
try:
pickle.dumps(state)
except pickle.PicklingError as err:
print(type(err).__name__, "while pickling; the notes say where:")
for note in err.__notes__:
print(" " + note)
# An open file, a lock and a generator belong to the running program: pickle refuses them.
with open("save.pkl", "rb") as f:
for thing in [f, threading.Lock(), (n * n for n in range(3))]:
try:
pickle.dumps(thing)
except TypeError as err:
print("TypeError:", err)Output
protocol: 5 | size: 224 bytes equal: True | a new object: True types kept: Player tuple ['lantern', 'map'] shared object kept: True class saved by name: True PicklingError while pickling; the notes say where: when serializing dict item 'on_win' when serializing game.GameState state when serializing game.GameState object TypeError: cannot pickle 'BufferedReader' instances TypeError: cannot pickle '_thread.lock' object TypeError: cannot pickle 'generator' object
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 game_state.py
game/game.py
"""The game's classes, in a module of their own: pickle saves a class by its module and name."""
from dataclasses import dataclass, field
@dataclass
class Player:
name: str
level: int
items: set = field(default_factory=set)
@dataclass
class GameState:
turn: int
players: list
scores: dict
board: tupleRuns on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
- The object comes back equal but new:
loaded == state, yet it is a different object, built from the bytes. - Types survive, unlike in JSON: the players are
Playerobjects again, the board is still a tuple of tuples and the items are a set. - Shared objects stay shared.
state.leaderandstate.players[0]were one object, and after loading they are one object again. Pickle remembers what it has already written and writes a reference the second time. - Classes and functions are saved by name, as their module and qualified name, never as code. Loading imports
game.GameStateagain, so the class must exist under the same name when the data is read. That is why the classes sit in a module: rename or move a class and old pickles stop loading.
The third point is the easiest to miss. Step through a smaller round trip: Turning data into another form and back, such as a dict into JSON text and back again. A round trip that loses nothing gives an equal value; through JSON, tuples come back as lists and keys as strings. More in “JSON: load, dump and custom types”, a list that
holds the same list twice, and watch which names point at one list after pickle.loads():
Program · shared_list.py
import pickle
asha = ["Asha", 4]
team = [asha, asha]
back = pickle.loads(pickle.dumps(team))
print(back == team, back is team)
first, second = back
print(first is second, first is asha)
first[1] = 5
print(back)
print(team)Variables
- changed in this step
Printed so far
Nothing printed yetTrue False True False [['Asha', 5], ['Asha', 5]] [['Asha', 4], ['Asha', 4]]
Every step was recorded from a run with Python 3.14.8 on macOS 26 arm64.
The drawing shows the variables when the program ends, and the program's 11 steps are listed below.
Change the program and run it here. Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Every step as text
- Start. No line has run yet, so there are no variables. Python starts with line 1.
- Line 1.
import pickleloadspickle. - Line 3.
asha = ["Asha", 4]makesashaa list of 2 items. - Line 4.
team = [asha, asha]makesteama list of 2 items. team holds one list twice. Both of its items are the list asha itself, not two copies of it. - Line 5.
back = pickle.loads(pickle.dumps(team))setsbackto a new list with the same items asteam,[['Asha', 4], ['Asha', 4]]. Key point dumps() wrote the inner list once and, the second time, a short reference to it. loads() built one new list from the bytes and put it in both places of back. - Line 6.
print(back == team, back is team)printsTrue False. back equals team item for item, but it is a new list built from the bytes, so back is team is False. - Line 7.
first, second = backmakesfirsta second name for the listsecondnames: one list, two names, not a copy and makesseconda second name for the listfirstnames: one list, two names, not a copy. - Line 8.
print(first is second, first is asha)printsTrue False. first and second are one list, as the two items of team were. It is not asha, though, because the round trip made a new one. - Line 9.
first[1] = 5changesfirst: it is now['Asha', 5]. Changing the list through first changes both items of back. team and asha still hold 4. - Line 10.
print(back)prints[['Asha', 5], ['Asha', 5]]. - Line 11.
print(team)prints[['Asha', 4], ['Asha', 4]]. That was the last line, so the program ends. The copy keeps its shared objects shared, and it shares nothing with the original.
first and second are one list, as the two items of team were, so the change made through first shows in both
items of back, while team still holds 4. The pickler’s
The record a pickler keeps of the objects it has already written. When it meets one again, it writes a short reference instead of a second copy, so shared and recursive objects come back shared.
did it: it wrote the inner list once and a reference to it the second time. Press Edit, change the fourth line to
team = [asha, ["Asha", 4]] and run it again: now the two lists are separate before and after the round trip, and
first is second prints False.
The end of the game example shows what cannot be pickled. A lambda has no name that loading could import later,
because every lambda is called <lambda>, so pickle.dumps() raises PicklingError. Since Python 3.14 the error
carries notes that say where the problem is, here the dictionary item on_win inside the GameState object, which
saves a long search in a big object. An open file, a lock and a generator belong to the running program, and pickle
refuses each of them with a TypeError, as the last three lines show.
Checkpoint
Check: What a round trip gives back
The protocol decides which Pythons can read a pickle
The first line of the game example printed protocol: 5. The protocol is the version of the pickle format, and
each new one added to the format, so a file written with a protocol can be read only by a Python that knows it. Walk
through the six:
- Protocol 0
The first format, in readable characters
The oldest form writes its pickles as characters a person can read, so a small one can be followed by eye. Every Python, old or new, loads it.
- Protocol 1
The first binary form
Values are written as binary data instead of readable characters. Like protocol 0, it loads in old and new Pythons alike.
- Protocol 2
Python 2.3: new-style classes
Python 2.3 added it so that objects of new-style classes, the only kind of class Python 3 has, pickle far more efficiently. PEP 307 lists the changes.
- Protocol 3
Python 3.0: bytes objects
Python 3.0 added opcodes for bytes objects. Python 2 knows none of them, so it cannot load these files. Python 3.0 to 3.7 wrote this protocol unless told otherwise.
- Protocol 4
Python 3.4: frames and very large objects
PEP 3154 added frames, which spare the unpickler many small reads, opcodes for strings and bytes over 4 GiB, and STACK_GLOBAL and MEMOIZE, which the Fraction pickle of the next section uses. Python 3.8 to 3.13 wrote it by default.
- Protocol 5
Python 3.8: out-of-band buffers the default now
PEP 574 added a way to hand large buffers, such as array data, to the program beside the pickle stream instead of copying them into it. Python 3.14 made it the default, so a file written with the default needs Python 3.8 or newer.
The timeline has 6 events, from Protocol 0 to Protocol 5.
Python 3.14 changed the default protocol to 5. Protocol 5 was added in Python 3.8, so files written with the default
can be read by 3.8 and newer; pass protocol=4 to dump() or dumps() for older readers. pickle.DEFAULT_PROTOCOL
says what the running Python writes by default and pickle.HIGHEST_PROTOCOL the newest it can write. They may
differ, as they did from Python 3.8 to 3.13, when the default was 4 and the highest 5.
Checkpoint
Check: Protocols and the Pythons that read them
A pickle is a program, and loading runs it
A pickle is not a passive description of data. It is a short program for a small stack machine inside the pickle
module, and pickletools.dis() shows its instructions, called opcode: One instruction of a small virtual machine, written as a byte and sometimes followed by an argument. A pickle is a sequence of opcodes, such as STACK_GLOBAL and REDUCE, that the unpickler carries out in order.. Here is a harmless
Fraction, disassembled:
import io
import pickle
import pickletools
from decimal import Decimal
from fractions import Fraction
# A pickle is a small program for a stack machine. Disassembling one shows its instructions.
data = pickle.dumps(Fraction(3, 4), protocol=5)
pickletools.dis(data, annotate=0)
# STACK_GLOBAL imports fractions.Fraction and REDUCE calls it with the arguments before it. Loading a
# pickle can name any importable function this way, which is why the documentation says it is not secure.
print(pickle.loads(data))
class AllowListUnpickler(pickle.Unpickler):
"""Refuses every global except the ones this program expects."""
ALLOWED = {("fractions", "Fraction")}
def find_class(self, module, name):
if (module, name) in self.ALLOWED:
return super().find_class(module, name)
raise pickle.UnpicklingError(f"refusing to load {module}.{name}")
for value in [{"marks": [71, 88], "rank": (2, 5)}, Fraction(3, 4), Decimal("2.50")]:
blob = pickle.dumps(value)
try:
print("loaded:", AllowListUnpickler(io.BytesIO(blob)).load())
except pickle.UnpicklingError as err:
print("refused:", err)Output
0: \x80 PROTO 5
2: \x95 FRAME 34
11: \x8c SHORT_BINUNICODE 'fractions'
22: \x94 MEMOIZE (as 0)
23: \x8c SHORT_BINUNICODE 'Fraction'
33: \x94 MEMOIZE (as 1)
34: \x93 STACK_GLOBAL
35: \x94 MEMOIZE (as 2)
36: K BININT1 3
38: K BININT1 4
40: \x86 TUPLE2
41: \x94 MEMOIZE (as 3)
42: R REDUCE
43: \x94 MEMOIZE (as 4)
44: . STOP
highest protocol among opcodes = 4
3/4
loaded: {'marks': [71, 88], 'rank': (2, 5)}
loaded: 3/4
refused: refusing to load decimal.Decimal
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 what_loading_does.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Step through the first half of that output, the 15 opcodes that rebuild Fraction(3, 4), and watch the stack, the
memo and the two steps where loading imports something and calls it:
The drawing shows the whole scene; its 8 steps are listed below.
Every step as text
- PROTO 5 and FRAME 34. The first two opcodes are a header. PROTO 5 names the protocol, and FRAME 34 says that the next 34 bytes form one frame, which the unpickler can fetch in one read.
- Push 'fractions'. SHORT_BINUNICODE pushes the string 'fractions' onto the stack. MEMOIZE then stores the top of the stack in the memo as entry 0, so a later opcode could refer to it.
- Push 'Fraction'. The same two opcodes push the string 'Fraction' and store it as entry 1. Two names now wait on the stack, and both of them came from the data.
- STACK_GLOBAL imports. STACK_GLOBAL takes the two strings off the stack and calls find_class('fractions', 'Fraction'), which imports the module fractions if needed and returns its class Fraction. The class goes on the stack.
- Push 3 and 4. BININT1 pushes the numbers 3 and 4, and TUPLE2 packs the two top items into the tuple (3, 4), which MEMOIZE stores as entry 3.
- REDUCE calls. REDUCE takes the tuple and the class off the stack and calls Fraction(3, 4). Here the callable is a harmless class. In crafted bytes it can be any importable function, called with arguments the data chooses.
- STOP returns. STOP ends the program, and load() returns the top of the stack, the fraction that the example prints as 3/4.
- Why it matters. Every object pickle rebuilds from a class goes through the same two steps, a name taken from the data and then a call. That is how malicious data can run arbitrary code during unpickling, as the documentation warns.
SHORT_BINUNICODE pushes the strings 'fractions' and 'Fraction', STACK_GLOBAL imports fractions.Fraction,
TUPLE2 packs the numbers 3 and 4, and REDUCE calls the imported object with them. The last line of the
disassembly, highest protocol among opcodes = 4, says that this pickle needed nothing newer than protocol 4, even
though its header names protocol 5.
That mechanism is how pickle rebuilds objects of any class, and it is also the danger: the module and the name come from the data. A crafted pickle can name any importable function and give it any arguments. That is why the pickle documentation warns that malicious data can run any code while it is unpickled, and asks you to unpickle nothing an untrusted source could have written or anyone could have changed. Checking the object after loading comes too late, because the code runs while it loads.
Key idea
Loading a pickle carries out instructions from the bytes, including “import this name” and “call it with these arguments”, so whoever writes the bytes chooses what your program runs.
Checkpoint
Check: What loading a pickle runs
A restricted unpickler decides which names a pickle may load
The second half of the example uses the defence the documentation describes under “Restricting globals”. An
Unpickler subclass overrides find_class(), which loading calls whenever a pickle asks for a global, a class or a
function, and refuses everything outside an allowlist: A list of what is accepted, such as the URL schemes http and https; anything not on it is refused. Unlike a denylist, it also refuses values nobody thought of. More in “XML and HTML parsing without security holes”. Step through the three values the
example loads with it:
The drawing shows the whole scene; its 4 steps are listed below.
Every step as text
- Plain data. The dictionary {'marks': [71, 88], 'rank': (2, 5)} is written with opcodes for dicts, lists, tuples, strings and numbers. It names no global, so find_class() is never called, and it loads.
- Fraction. Fraction(3, 4) names fractions.Fraction. STACK_GLOBAL asks find_class(), the name is in ALLOWED, so super().find_class() returns the class and REDUCE builds 3/4.
- Decimal. Decimal('2.50') names decimal.Decimal, which is not in ALLOWED. find_class() raises UnpicklingError, so loading stops before REDUCE could call anything.
- What it cannot do. An allowed class is still called with arguments from the data, so every name in the list must be safe to call with any arguments. The allowlist narrows what a pickle can reach. It does not make a stranger's pickle safe.
Plain data with no globals loads, the allowed Fraction loads, and decimal.Decimal is refused before anything could
call it. An allowlist narrows what a pickle can reach, but every allowed class must itself be safe to call with any
arguments, which is hard to judge. Treat it as a second line of defence, never as permission to load pickles from
strangers.
Security
Machine-learning model files are often pickles. The PyTorch documentation says that torch.load() uses an unpickler
and that you must never load data from an untrusted source; its weights_only mode restricts loading to tensors,
plain types and dictionaries. Formats built for the purpose, such as safetensors, store tensors without pickle at
all. Prefer them for any model you download.
Checkpoint
Check: Restricting globals with find_class()
An HMAC tag refuses changed bytes before they are unpickled
When pickles must leave the program and come back, for example in a cache or in files on a shared disk, sign them, as the pickle documentation itself suggests. An HMAC: A tag computed from data and a secret key with a hash function such as SHA-256, as hmac.digest() does. Without the key nobody can make a tag that matches changed data, so a matching tag shows the data is what a key holder signed. is a tag computed from the data and a key: anyone can compute a hash, but without the key nobody can compute a tag that matches changed data. The program stores the tag beside the data and, when loading, recomputes it and compares before unpickling anything:
Sign when saving, check before unpickling
Text description of the diagram
The flowchart follows one object from top to bottom, through saving and then loading.
- The program pickles a Python object into bytes with pickle.dumps().
- It computes an HMAC-SHA256 tag of those bytes with hmac.digest(), using a key that only the program knows.
- The file holds the tag followed by the pickled data.
- When loading, the program computes the HMAC of the data part again and compares it with the stored tag.
- If they match, the data is exactly what the program wrote, and only then is it passed to pickle.loads(). If they do not match, the program raises ValueError and never unpickles the data.
import hashlib
import hmac
import pickle
import secrets
KEY = secrets.token_bytes(32) # in a real program: a key from configuration, never stored with the data
TAG_SIZE = hashlib.sha256().digest_size # 32 bytes
def dumps_signed(obj, key):
data = pickle.dumps(obj)
return hmac.digest(key, data, "sha256") + data
def loads_signed(blob, key):
tag, data = blob[:TAG_SIZE], blob[TAG_SIZE:]
expected = hmac.digest(key, data, "sha256")
if not hmac.compare_digest(tag, expected): # constant-time comparison, never ==
raise ValueError("signature check failed: the data was changed or the key is wrong")
return pickle.loads(data) # only reached for bytes this program signed
blob = dumps_signed({"user": "asha", "cart": ["tea", "rice"]}, KEY)
print("signed blob:", len(blob), "bytes, of which", TAG_SIZE, "are the tag")
print("intact:", loads_signed(blob, KEY))
tampered = bytearray(blob)
tampered[-5] ^= 0x01 # flip one bit in the pickled data
for label, data, key in [("one bit flipped", bytes(tampered), KEY), ("another key", blob, secrets.token_bytes(32))]:
try:
loads_signed(data, key)
except ValueError as err:
print(f"{label}: {err}")Output
signed blob: 86 bytes, of which 32 are the tag
intact: {'user': 'asha', 'cart': ['tea', 'rice']}
one bit flipped: signature check failed: the data was changed or the key is wrong
another key: signature check failed: the data was changed or the key is wrong
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 signed_pickle.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Step through the run: one blob is signed, then loaded three times, intact, with one bit flipped and with another key:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- Sign. dumps_signed() pickles the dictionary into 54 bytes, computes their HMAC-SHA256 tag with the key, 32 bytes, and stores the tag and the data together, 86 bytes in all.
- Check the tag. loads_signed() splits the blob after byte 32, computes the HMAC of the 54 data bytes again and compares the two tags with compare_digest(). They match.
- Then unpickle. Only now do the 54 bytes reach pickle.loads(), which gives back {'user': 'asha', 'cart': ['tea', 'rice']}.
- One bit flips. The program flips one bit near the end of the stored data, as anyone who can write to the store could.
- Refused. The HMAC of the changed data no longer matches the stored tag, so loads_signed() raises ValueError, and pickle.loads() is never called.
- Another key. The intact blob, checked with a different random key, fails the same way. Without the key nobody can make a tag that matches, so nothing reaches pickle.loads().
hmac.digest(key, data, "sha256")gives a 32-byte tag. The blob is the tag followed by the 54 bytes of the pickle.- One flipped bit fails the check, and so does the right data with another key. In both cases
pickle.loads()never runs. hmac.compare_digest()compares the tags. The hmac documentation recommends it over==to reduce the risk of timing attacks:==stops at the first byte that differs, so its time hints at how much of a guessed tag is right, whilecompare_digest()does not stop early.- The key is the whole defence. Keep it out of the data, out of the code repository and away from anyone who can write the data, for example in an environment variable: A name with a text value that a process receives from the program that started it, such as APP_PORT=9000. Python reads them from os.environ, always as str, and every process it starts gets a copy of them. More in “Config files, environment variables and secrets” or a secrets manager: A service built to store credentials such as passwords and API keys and hand each program only the ones it may read, at run time, so they stay out of code, config files and the environment. More in “Config files, environment variables and secrets”. Anyone who has the key can sign anything.
Signing proves the bytes came from a holder of the key; it does not make pickle a good exchange format. For data that crosses a trust boundary: The line between data your program made and protects and data that someone else could have written or changed, such as an upload, a shared cache or a folder others can write to. Data that crosses it is read with care, in a format that cannot run code., a format that cannot run code is the better fix, and JSON is usually the right one.
Checkpoint
Check: Signing pickles with HMAC
A shelf hands out copies, so store your changes back
shelve.open(filename) returns an object that behaves like a dictionary whose keys are strings and whose values are
any picklable objects, stored in a database file through the dbm module. It suits small programs that need to keep
some state between runs without designing tables:
import shelve
from pathlib import Path
for old in Path(".").glob("progress*"): # start from an empty shelf on every run
old.unlink()
# A shelf is a dictionary on disk: string keys, pickled values.
with shelve.open("progress") as db:
db["asha"] = {"lessons_done": ["lists", "dicts"], "streak": 3}
db["ravi"] = {"lessons_done": ["lists"], "streak": 1}
with shelve.open("progress") as db:
print(sorted(db.keys()), "->", db["asha"]["streak"])
db["asha"]["lessons_done"].append("sets") # changes a copy that is thrown away
print("after append:", db["asha"]["lessons_done"])
record = db["asha"] # the fix: read, change, store back
record["lessons_done"].append("sets")
db["asha"] = record
print("after storing back:", db["asha"]["lessons_done"])
with shelve.open("progress", writeback=True) as db: # caches what you read and writes it all at close()
db["ravi"]["lessons_done"].append("dicts")
with shelve.open("progress", flag="r") as db: # read-only
print("with writeback:", db["ravi"]["lessons_done"])Output
['asha', 'ravi'] -> 3 after append: ['lists', 'dicts'] after storing back: ['lists', 'dicts', 'sets'] with writeback: ['lists', 'dicts']
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 shelf_store.py
This example shows recorded output only, because the Python that runs in the browser leaves out the dbm module that
shelve needs. The output shows the classic surprise: db["asha"]["lessons_done"].append("sets") changed a copy. Step
through what each line does to the shelf and to the file:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- Store a record. db['asha'] = {…} pickles Asha's record and stores the bytes under the key 'asha'. Her list holds lists and dicts.
- A read is a copy. db['asha'] unpickles the stored bytes into a new dictionary. The shelf hands it over and keeps no link to it.
- The append is lost. db['asha']['lessons_done'].append('sets') changes that copy, which nothing keeps. The next read unpickles the stored bytes again and prints ['lists', 'dicts'].
- Store it back. record = db['asha'], an append to record, then db['asha'] = record pickles the changed copy and stores it, so the next read prints ['lists', 'dicts', 'sets'].
- writeback=True. Opened with writeback=True, the shelf keeps every value it hands out in a cache, so db['ravi']['lessons_done'].append('dicts') changes the cached object.
- close() writes back. When the shelf closes, it pickles every cached entry and stores it, so Ravi's list now holds lists and dicts. The cost is memory for the cache and a slower close().
Each db["asha"] unpickles a fresh object, and the shelf cannot know that you changed it, so the change is lost.
Either read the value, change it and assign it back, or open the shelf with writeback=True, which keeps every value
you read in memory and writes them all back when the shelf closes, at the cost of memory and a slow close().
Changing a shelf value in place
db["asha"]["lessons_done"].append("sets") changes a copy that is thrown away, and the shelf still holds
['lists', 'dicts']. Read the record into a name, change it, then store it back with db["asha"] = record, or open
the shelf with writeback=True.
Two more rules from the documentation: a shelf has the same trust rules as pickle, so never open one from an untrusted source, and it does not support several programs writing at the same time. For shared data or concurrent writers, use SQLite.
Checkpoint
Check: Keeping changes in a shelf
Choose the format by who writes the data and who reads it
Four formats have come up in this module: pickle, JSON, CSV and SQLite. Four questions decide between them: where the data comes from, who reads it, how it changes and which types must come back. Answer them in the table for a job of your own, and it marks the formats that fit:
Answer the questions above: the table then marks what fits your case and says why the rest does not.
| Format | Data from strangers | Other languages | Python types kept | Change one record |
|---|---|---|---|---|
| pickle Suits: short-lived data your program wrote, such as a cache or a checkpoint | Poor loading can run code that the bytes name | Poor Python only | Good almost all, your own classes too | Poor rewrite the whole file |
| JSON Suits: settings, API data and anything stored or shared | Good json.load() builds only dicts, lists, strings, numbers, booleans and None | Good every language | Mixed dict, list, str, numbers, bool and None; tuples come back as lists | Poor rewrite the whole file |
| CSV Suits: tables for people and spreadsheets | Good the reader builds rows of strings | Good every language, and spreadsheets | Poor strings only | Poor rewrite the whole file |
| SQLite Suits: records you query, update and share between programs | Mixed runs no Python code, but SQLite's security page says to treat a file others wrote as suspect | Good every language with an SQLite library | Mixed numbers, text, bytes and NULL | Good update in place, in a transaction |
How the answers decide
- “Can the data come from outside your program, such as an upload, an API or a folder others write to?” Yes, others can write it: rules out pickle. Loading a pickle follows instructions in the bytes, so whoever writes them chooses what your program runs. The pickle documentation says never to unpickle such data.
- “Can the data come from outside your program, such as an upload, an API or a folder others write to?” Yes, others can write it: adds a caveat to SQLite. A database file others could write is not Python code, but SQLite's own security page asks you to treat it as suspect and lists the precautions.
- “Must other languages, other programs or people with a spreadsheet read it?” Yes: rules out pickle. The pickle format is specific to Python, and loading it needs the same classes importable under the same names.
- “Do you search it, change single records or have more than one writer?” Yes: adds a caveat to JSON and CSV. Every change rewrites the whole file, and two programs writing at once can lose each other's changes. SQLite updates one record in a transaction.
- “Do you search it, change single records or have more than one writer?” Yes: rules out pickle. A pickle is written and read whole, by one program; it has no way to search or change one record.
- “Must your own classes, sets or tuples come back exactly as they were?” Yes: adds a caveat to JSON, CSV and SQLite. These formats give back plain values. Convert your objects to dicts and back, as the JSON lesson does, or keep pickle for data that never leaves your program.
A rule of thumb follows from the table. Use JSON by default for data that is stored or sent. Use CSV for tables that people open in a spreadsheet. Use SQLite, the next lesson’s subject, as soon as you search, update single records or have more than one writer. Keep pickle for data that never leaves your control and does not need to outlive your code, such as a cache of computed results or a checkpoint of a long job.
Checkpoint
Check: Choosing a format
Practise: save program state as a signed pickle
The exercise asks for save_state(), which writes the HMAC tag and then the pickled bytes, and load_state(), which
refuses a short file, a changed byte or another key before anything is unpickled. Its sample tests run in your
browser:
Exercise · Medium · Python
Save program state as a signed pickle
Write two functions in state_store.py. Keep the names and parameters: the tests import them.
save_state(path, obj, key) pickles obj and writes one file: first the 32-byte HMAC-SHA256 tag of the pickled bytes, computed with key (a bytes value), then the pickled bytes themselves.
load_state(path, key) reads such a file and returns the object, but only after checking the tag:
- Raise
ValueErrorwhen the file is shorter than 32 bytes, when the tag does not match the data, or when the key is not the one the file was saved with. - Never call
pickle.loads()on data whose tag has not been checked: a file with a wrong tag must raiseValueErroreven when its data part is not a valid pickle at all. - Compare tags with
hmac.compare_digest(), not==.
Starter code · state_store.py
import hmac
import pickle
def save_state(path, obj, key):
"""Write the HMAC-SHA256 tag of the pickled obj, then the pickled bytes, to path."""
with open(path, "wb") as f:
f.write(pickle.dumps(obj))
def load_state(path, key):
"""Return the object saved at path; raise ValueError unless its tag matches."""
with open(path, "rb") as f:
return pickle.loads(f.read())The sample tests · test_state_store.py
import hmac
import tempfile
from pathlib import Path
from state_store import load_state, save_state
KEY = b"k" * 32
STATE = {"turn": 12, "players": ("Asha", "Ravi"), "seen": {3, 5, 8}, "notes": ["fog", None]}
def saved(obj=STATE, key=KEY):
path = Path(tempfile.mkdtemp()) / "state.bin"
save_state(path, obj, key)
return path
def expect_value_error(path, key=KEY):
try:
load_state(path, key)
except ValueError:
return
assert False, "load_state accepted data it should refuse"
def test_round_trip():
"""an object comes back equal, with its tuple and set intact"""
assert load_state(saved(), KEY) == STATE
def test_file_layout():
"""the file is the 32-byte HMAC-SHA256 tag of the data, then the data"""
blob = saved().read_bytes()
assert blob[:32] == hmac.digest(KEY, blob[32:], "sha256")
def test_flipped_data_byte():
"""one changed byte in the data is refused"""
path = saved()
blob = bytearray(path.read_bytes())
blob[-3] ^= 0x01
path.write_bytes(bytes(blob))
expect_value_error(path)
def test_flipped_tag_byte():
"""one changed byte in the tag is refused"""
path = saved()
blob = bytearray(path.read_bytes())
blob[0] ^= 0x80
path.write_bytes(bytes(blob))
expect_value_error(path)
def test_wrong_key():
"""a file saved with another key is refused"""
expect_value_error(saved(), key=b"x" * 32)
def test_short_file():
"""a file too short to hold a tag is refused"""
path = Path(tempfile.mkdtemp()) / "short.bin"
path.write_bytes(b"0123456789")
expect_value_error(path)
def test_checks_before_unpickling():
"""a wrong tag in front of bytes that are not a pickle still gives ValueError"""
path = Path(tempfile.mkdtemp()) / "junk.bin"
path.write_bytes(b"\x00" * 32 + b"this is not a pickle")
expect_value_error(path)A hint
hmac.digest(key, data, "sha256") returns the 32-byte tag. In load_state(), split what you read with blob[:32] and blob[32:], compute the tag of the second part again, and compare the two with hmac.compare_digest() before anything is unpickled.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
The practice below mixes questions on the game example, the opcode figure, the signed blob and the shelf. It begins with whichever of those your checkpoint answers left shakiest, and explains every pick on the spot:
Practice that adapts to you
Practise this lesson
18 questions about 7 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
Why is it unsafe to unpickle data from an untrusted source?
Because a pickle is a program, not just data. Loading it runs instructions that the bytes contain, and two of them,
STACK_GLOBAL (or GLOBAL in older protocols) and REDUCE, import an object named in the data and call it with
arguments from the data. That is how pickle rebuilds instances of any class, and it means crafted bytes can name any
importable function, so pickle.loads() can run code with all the permissions of your program. The pickle
documentation says so in its first warning: only unpickle data you trust. Checking the data after loading is too
late, because the code runs during loading. The fixes are to use a format that cannot run code, such as JSON, for
anything that crosses a trust boundary, or to sign pickles your program writes with HMAC and verify the tag before
loading.
Key takeaways
pickle.dumps()/loads()anddump()/load()save almost any object, with types, sets, tuples and shared references intact, in files opened in binary mode; lambdas, open files, locks and generators are refused.- Classes and functions are pickled by module and name, so they must be importable under the same name when you load, and since Python 3.14 the default protocol is 5, which Python 3.8 and newer can read.
- Loading a pickle carries out its opcodes, and
STACK_GLOBALandREDUCEimport and call whatever the data names, so never unpickle untrusted data; afind_class()allowlist only narrows what a pickle can reach. - Sign pickles that leave your program with
hmac.digest(), check the tag withhmac.compare_digest()beforepickle.loads(), and keep the key away from the data. - A shelf hands out unpickled copies, so assign changed values back or open it with
writeback=True, and never share it between writers. - Default to JSON for stored or exchanged data, CSV for tables, SQLite for records you update or share, and pickle only for short-lived data you control.
Terms in this lesson
- HMAC
- A tag computed from data and a secret key with a hash function such as SHA-256, as hmac.digest() does. Without the key nobody can make a tag that matches changed data, so a matching tag shows the data is what a key holder signed.
- opcode
- One instruction of a small virtual machine, written as a byte and sometimes followed by an argument. A pickle is a sequence of opcodes, such as STACK_GLOBAL and REDUCE, that the unpickler carries out in order.
- trust boundary
- The line between data your program made and protects and data that someone else could have written or changed, such as an upload, a shared cache or a folder others can write to. Data that crosses it is read with care, in a format that cannot run code.
References
- pickle: Python object serialization (Python Software Foundation)
- pickle: data stream format (the six protocols) (Python Software Foundation)
- pickle: restricting globals (Python Software Foundation)
- PEP 3154: Pickle protocol version 4 (Python Software Foundation)
- pickletools: tools for pickle developers (Python Software Foundation)
- shelve: Python object persistence (Python Software Foundation)
- hmac: keyed-hashing for message authentication (Python Software Foundation)
- json: JSON encoder and decoder (Python Software Foundation)
- What's New In Python 3.14: pickle (Python Software Foundation)
- Pyodide: WebAssembly constraints (removed modules) (Pyodide project)
- Defense Against The Dark Arts (untrusted SQLite database files) (SQLite)
- torch.load (PyTorch documentation) (PyTorch Foundation)
- Safetensors documentation (Hugging Face)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress