MySmartCoPilot
Ask Passes Your pass 7 days left 1 day left Last day

Your country

Tools that support it use your country for local currency, number formats, units and paper size. Your choice is saved only in this browser.

Python  Module 8 – Files, paths and data formats

Config files, environment variables and secrets

Read TOML settings with tomllib and INI files with configparser, take settings and secrets from environment variables, and keep keys out of code and logs.

  • Intermediate
  • 25 minutes
  • Examples run with Python 3.14.8 and Pyodide 314.0.7
  • By MySmartCoPilot

What you will learn

  • Read TOML configuration with tomllib and INI files with configparser
  • Convert settings read from environment variables to the right type
  • Combine defaults, config files, environment variables and flags in a fixed order
  • Prevent credentials from leaking into source code, logs, error messages and child processes

Before you start

What you’ll learn · Before you start

What you will learn

  • Read TOML configuration with tomllib and INI files with configparser
  • Convert settings read from environment variables to the right type
  • Combine defaults, config files, environment variables and flags in a fixed order
  • Prevent credentials from leaking into source code, logs, error messages and child processes

Before you start

On this page

The same program runs on a laptop, a test server and a production server, and on each of the three it needs a different database address, port and API key. Written into the code, every change is a new commit, and the key is readable by everyone who can read the repository. Python reads such settings from outside the code instead: TOML files with tomllib, INI files with configparser, environment variables from os.environ, and flags from the command line.

Here you will read each kind, combine them in a fixed order so that you always know which value wins, and keep secrets out of code, logs, error messages and the programs you start.

tomllib turns a TOML file into typed values

Is 8080 in a settings file the number 8080 or the text “8080”? In an INI file it is always text; in TOML, the format of pyproject.toml, the way it is written decides. TOML writes keys and values with their types and groups keys under [table] headings, and tomllib, in the standard library since Python 3.11, turns each value into the matching Python type:

Settings from TOML, with their types Python · toml_settings.py 
import tomllib

CONFIG = """
app_name = "Marks Portal"
debug = false
port = 8080
timeout = 2.5
allowed_hosts = ["localhost", "marks.example.com"]

[database]
host = "db.internal"
port = 5432
"""

settings = tomllib.loads(CONFIG)  # loads: from a str
for key, value in settings.items():
    print(f"{key:<14} {type(value).__name__:<6} {value!r}")
print("database port + 1 =", settings["database"]["port"] + 1)  # already an int

with open("app.toml", "w", encoding="utf-8") as f:
    f.write(CONFIG)
with open("app.toml", "rb") as f:  # load: from a file opened in binary mode
    print("same from the file:", tomllib.load(f) == settings)

try:
    with open("app.toml", encoding="utf-8") as f:  # text mode is refused
        tomllib.load(f)
except TypeError as err:
    print("TypeError:", err)

try:
    tomllib.loads('port = 80\nhost = localhost\n')  # strings need quotes in TOML
except tomllib.TOMLDecodeError as err:
    print(f"TOMLDecodeError: {err.msg} at line {err.lineno}, column {err.colno}")

Output

app_name       str    'Marks Portal'
debug          bool   False
port           int    8080
timeout        float  2.5
allowed_hosts  list   ['localhost', 'marks.example.com']
database       dict   {'host': 'db.internal', 'port': 5432}
database port + 1 = 5433
same from the file: True
TypeError: File must be opened in binary mode, e.g. use `open('foo.toml', 'rb')`
TOMLDecodeError: Invalid value at line 2, column 8

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 toml_settings.py

Each line of the output gives a key, the type its value arrived as, and the value. The table puts the conversions of the tomllib documentation beside the keys of the example:

TOML value Python type In the example
a string in quotes str app_name = "Marks Portal"
true or false bool debug = false
an integer int port = 8080
a float float timeout = 2.5
an array […] list allowed_hosts
a table […] dict [database]
a date, a time, or both datetime.date, datetime.time, datetime.datetime none

So the database port is an int at once, and adding 1 to it gives 5433. Press Edit, put quotes around 8080 and run the program again: the port line now says str, because a value in quotes is a TOML string.

tomllib.loads() reads a string and tomllib.load() a file, which must be opened in binary mode. The TOML specification requires every TOML file to be UTF-8: The Unicode encoding that writes each code point as 1 to 4 bytes: 1 for ASCII, 2 for é, 3 for Devanagari and 4 for most emoji. Any ASCII text is also valid UTF-8, byte for byte. More in “Text vs bytes: Unicode, encodings, bytearray”, so the parser decodes the bytes itself and refuses a file opened in text mode, as the TypeError shows. A syntax error raises tomllib.TOMLDecodeError; since Python 3.14 its msg, lineno and colno attributes point at the mistake, here the value localhost, which needed quotes.

Opening a TOML file in text mode

tomllib.load(open("app.toml", encoding="utf-8")) raises TypeError, because load() takes bytes. Open the file with "rb", or read the text yourself and pass it to tomllib.loads().

tomllib only reads. To write TOML, its documentation points to two packages from outside the standard library: tomli-w, which writes data, and tomlkit, which also keeps the comments and layout of a file it edits.

Checkpoint

Check: Typed values from TOML

  1. Question 1 of 2 What does this print?

    Read the code, then choose one answer.

    import tomllib

data = tomllib.loads('ports = [8080, 8081]\nname = "web"\n')
print(type(data["ports"]).__name__, type(data["ports"][0]).__name__)
    Show the answer

    Answer: list int

    tomllib converts TOML types to Python types: an array to a list, an integer to an int. So data["ports"] is [8080, 8081], a list of two ints.

  2. Question 2 of 2 Why does tomllib.load(open("app.toml", encoding="utf-8")) raise TypeError?

    Choose one answer.

    Show the answer

    Answer: load() needs a file opened in binary mode ("rb"), because it decodes the UTF-8 itself

    tomllib.load() takes a binary file object and decodes its bytes as UTF-8, as the TOML specification requires. Open the file with "rb", or read the text yourself and pass it to tomllib.loads().

configparser reads INI files as text, with its own rules

INI files group key = value lines under [section] headings, and many older tools and Windows programs use them. configparser reads them, and four of its rules surprise people. The example meets all four:

An INI file and its surprises Python · ini_config.py 
import configparser

INI = """
[DEFAULT]
timeout = 30

[database]
Host = db.internal
port = 5432
password = s3cr%t

[mail]
enabled = yes
"""

config = configparser.ConfigParser()
config.read_string(INI)
print("sections:", config.sections())  # DEFAULT is not listed, but every section sees it
db = config["database"]
print("host:", db["host"])  # keys are not case-sensitive: Host and host are the same key
print("port:", repr(db["port"]), "->", db.getint("port"))  # values are strings until you convert them
print("mail enabled:", config.getboolean("mail", "enabled"))  # yes/no, on/off, true/false, 1/0
print("mail timeout:", config["mail"]["timeout"])  # inherited from DEFAULT

try:
    db["password"]  # % starts an interpolation such as %(host)s
except configparser.InterpolationSyntaxError as err:
    print("InterpolationSyntaxError:", err)

raw = configparser.ConfigParser(interpolation=None)  # no interpolation: % is just a character
raw.read_string(INI)
print("password length:", len(raw["database"]["password"]))

Output

sections: ['database', 'mail']
host: db.internal
port: '5432' -> 5432
mail enabled: True
mail timeout: 30
InterpolationSyntaxError: '%' must be followed by '%' or '(', found: '%t'
password length: 6

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 ini_config.py

Step through the lookups the example makes, from the key the code asks for to the text that comes back:

Interactive · stage by stage How configparser finds a value in an INI file Step through the lookups of ini_config.py. Each step follows one key from the code that asks for it to the text that comes back.
How configparser finds a value in an INI file, step 7 of 7. Showing Your code asks for a key (db['password']), The key, in lower case, The section's own keys, Then [DEFAULT], Interpolation (off: % is a character), A str comes back ('yes', then True).found in the sectionYour code asks for a keydb['password']The key, in lower caseoptionxform(): Host is hostThe section's own keys[database], [mail]Then [DEFAULT]every section inherits itInterpolationoff: % is a characterA str comes back'yes', then True

The drawing shows the whole scene; its 7 steps are listed below.

Every step as text
  1. Host is host. db['host'] asks the [database] section for host. The file wrote Host, but optionxform() turns every key into lower case, both when the file is read and when your code asks, so the two are the same key.
  2. Found in the section. [database] has the key, so the lookup skips [DEFAULT] and its value comes back as the text 'db.internal'. Section names keep their case: config['Database'] would raise KeyError.
  3. Numbers are text. db['port'] gives the text '5432'. Config parsers never guess types, so the number comes from getint('port'): 5432. getfloat() and getboolean() convert in the same way.
  4. Inherited from [DEFAULT]. mail['timeout'] is not among the keys of [mail]. The lookup goes on to [DEFAULT], whose keys every section inherits, and finds '30'. sections() never lists DEFAULT itself.
  5. yes is True. getboolean('mail', 'enabled') reads 'yes'. It accepts 1, yes, true and on for True and 0, no, false and off for False, in any case, and raises ValueError for any other word.
  6. % starts a reference. db['password'] holds s3cr%t. Interpolation runs when a value is read: %(host)s would be replaced by the value of host and %% means one percent sign, so a lone %t raises InterpolationSyntaxError.
  7. interpolation=None. With ConfigParser(interpolation=None), a percent sign is an ordinary character and the password reads back as it was written, 6 characters. Use it for files that hold secrets or paths.

  • Every value is a string. Config parsers never guess types. getint(), getfloat() and getboolean() convert, and getboolean() accepts 1, yes, true and on for True and 0, no, false and off for False, in any case.
  • [DEFAULT] is not a section like the others. sections() does not list it, but every section inherits its keys, which is why mail has a timeout.
  • Keys ignore case, so Host and host are the same key, while section names keep theirs: config["Database"] raises KeyError.
  • % starts an interpolation (configparser): Replacing a reference such as %(host)s inside an INI value with another value of the same section or of [DEFAULT], when the value is read. A lone % raises InterpolationSyntaxError; interpolation=None turns it off., a reference to another value such as %(host)s, so a lone % raises InterpolationSyntaxError when the value is read. Pass interpolation=None when values may contain %, which is the safe choice for anything that holds secrets or paths.

Two more details matter when files are optional or written back. config.read(paths) silently skips files that it cannot open and returns the list of files it did read. And since Python 3.14, write() raises configparser.InvalidWriteError instead of writing a key that it could not read back, such as a key that contains = or begins like a [section] heading.

Trusting read() to report a missing file

config.read(["app.ini"]) returns [] when app.ini is missing, and the program carries on with no settings. For a file you need, open it yourself and pass it to read_file(), so that a missing file raises FileNotFoundError, or check the list that read() returns.

Checkpoint

Check: INI files and configparser

  1. Question 1 of 2 What does this print?

    Read the code, then choose one answer.

    import configparser

config = configparser.ConfigParser()
config.read_string("[DEFAULT]\nretries = 3\n\n[mail]\nPort = 25\n")
print(config.sections(), config["mail"]["port"], config["mail"]["retries"])
    Show the answer

    Answer: ['mail'] 25 3

    sections() lists every section but DEFAULT. Keys ignore case, so config["mail"]["port"] finds Port and gives the text 25, and retries comes from [DEFAULT], which every section inherits.

  2. Question 2 of 2 An INI file holds password = 50%off. Reading config["db"]["password"] with a default ConfigParser() raises InterpolationSyntaxError. What is the best fix?

    Choose one answer.

    Show the answer

    Answer: Create the parser with ConfigParser(interpolation=None)

    With the default BasicInterpolation, % starts a reference such as %(host)s, and %% stands for one percent sign. For files that hold secrets or paths, interpolation=None is the safe choice; writing 50%%off also works.

Environment variables arrive as text

Every process gets a set of environment variable: A name with a text value that a process receives from the program that started it, such as APP_PORT=9000. Python reads them from os.environ, always as str, and every process it starts gets a copy of them. from the program that started it: the shell, a scheduler or a container platform. In Python they are in os.environ, a mapping from names to strings. A shell that runs APP_DEBUG=false python app.py means “debugging off”, and the classic bug reads it as on. Step through it and watch text, wrong and right:

Interactive · line by line A switch from the environment: bool() and the fix Step through the program. The lit line has just run; watch text, wrong and right, and what each one says about debugging.

Program · env_bool_bug.py

import os

# As if the shell had set it:
os.environ["APP_DEBUG"] = "false"
TRUE = {"1", "true", "yes", "on"}

text = os.environ["APP_DEBUG"]
wrong = bool(text)
word = text.strip().lower()
right = word in TRUE
print("bool(text):", wrong)
print("in TRUE:", right)

Variables

After line 12: TRUE is {'1', 'on', 'true', 'yes'}; text is 'false'; wrong is True; word is 'false'; right is False.TRUE'1''on''true''yes'text'false'wrongTrueword'false'rightFalse
  • changed in this step

Printed so far

bool(text): True
in TRUE: False

Every step was recorded from a run with Python 3.14.8 on macOS 26 arm64.

The drawing shows the variables when the program ends, and the program's 10 steps are listed below.

Line run 12
Next line end
Every step as text
  1. Start. No line has run yet, so there are no variables. Python starts with line 1.
  2. Line 1. import os loads os.
  3. Line 4. os.environ["APP_DEBUG"] = "false" runs.
  4. Line 5. TRUE = {"1", "true", "yes", "on"} makes TRUE a set of 4 items.
  5. Line 7. text = os.environ["APP_DEBUG"] sets text to 'false'. Key point text is the string 'false'. The environment holds text only, so os.environ gives back exactly what the shell wrote.
  6. Line 8. wrong = bool(text) sets wrong to True. The bug bool() asks only whether a string is empty. 'false' has five characters, so wrong is True and debugging would be switched on.
  7. Line 9. word = text.strip().lower() sets word to 'false'.
  8. Line 10. right = word in TRUE sets right to False. The fix Compared with the words you accept, the text gives the answer the shell meant. 'false' is not in TRUE, so right is False.
  9. Line 11. print("bool(text):", wrong) prints bool(text): True.
  10. Line 12. print("in TRUE:", right) prints in TRUE: False. That was the last line, so the program ends. A word in neither set, such as a typo, should be an error that names APP_DEBUG. The next program does that.

bool() asks only whether a string is empty, so bool("false") is True, and so are bool("0") and bool("no"). Comparing the text with the words you accept gives the answer the shell meant. A complete reader also handles a variable that is not set, and refuses a word it does not know with an error that names the variable, so that the person who set it knows what to fix:

Reading numbers and switches from the environment Python · env_vars.py 
import os

# A shell would set these before starting the program; the example sets them itself.
os.environ["APP_DEBUG"] = "false"
os.environ["APP_WORKERS"] = "4"

print(repr(os.environ["APP_WORKERS"]))  # always a str
print(os.environ.get("APP_REGION", "south-1"))  # a default for a variable that is not set
print("bool() of the text:", bool(os.environ["APP_DEBUG"]))  # any non-empty str is true!

TRUE, FALSE = {"1", "true", "yes", "on"}, {"0", "false", "no", "off"}


def env_bool(name, default):
    text = os.environ.get(name)
    if text is None:
        return default
    if text.strip().lower() in TRUE:
        return True
    if text.strip().lower() in FALSE:
        return False
    raise ValueError(f"{name} must be one of {sorted(TRUE | FALSE)}, not {text!r}")


def env_int(name, default):
    text = os.environ.get(name)
    try:
        return default if text is None else int(text)
    except ValueError:
        raise ValueError(f"{name} must be a whole number, not {text!r}") from None


print("debug:", env_bool("APP_DEBUG", default=False))
print("workers:", env_int("APP_WORKERS", default=1))
os.environ["APP_WORKERS"] = "four"
try:
    env_int("APP_WORKERS", default=1)
except ValueError as err:
    print("ValueError:", err)

Output

'4'
south-1
bool() of the text: True
debug: False
workers: 4
ValueError: APP_WORKERS must be a whole number, not 'four'

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 env_vars.py

os.environ["NAME"] raises KeyError when the variable is not set, and os.environ.get("NAME", default) returns the default instead, as APP_REGION shows. env_int() turns "four" into a ValueError that says which variable was wrong. Assigning to os.environ changes the environment of the running program and of the processes it starts afterwards; it does not change the shell that started the program.

Testing a switch with bool()

bool(os.environ.get("DEBUG")) is True for "false", "0" and "off". Compare the stripped, lower-case text with explicit sets such as {"1", "true", "yes", "on"} and {"0", "false", "no", "off"}, and raise an error for anything else.

Checkpoint

Check: Text from the environment

  1. Question 1 of 2 APP_REGION is not set. What do os.environ["APP_REGION"] and os.environ.get("APP_REGION", "south-1") do?

    Choose one answer.

    Show the answer

    Answer: The first raises KeyError, the second returns 'south-1'

    os.environ behaves like a dict of strings: indexing a missing name raises KeyError, and get(name, default) returns the default instead.

  2. Question 2 of 2 The shell started the program with CACHE=0, meaning "no cache". What does this print?

    Read the code, then choose one answer.

    import os

if bool(os.environ.get("CACHE")):
    print("cache on")
else:
    print("cache off")
    Show the answer

    Answer: cache on

    Environment variables are text, so CACHE is "0", and bool("0") is True. Compare the stripped, lower-case text with the words you accept, such as {"0", "false", "no", "off"} for off.

A fixed order decides which setting wins

The code’s default says port 8000, the config file port = 8080 and the environment APP_PORT=9000: which port does the program use? Real programs take settings from several places, so they need one rule for which place wins. A common order, from weakest to strongest: defaults written in the code, then the config file, then environment variables, then command-line flags. Each layer is more specific than the one before: the file is shared by every installation, the environment belongs to one machine or deployment, and a flag to one run. The diagram follows one setting, the port, through all four:

Four layers of settings searched in order: command-line flags, environment variables, the config file, and the defaults in the code.The program asksfor a setting1. Command-line flags--port=9100chosen for this one run2. Environment variablesAPP_PORT=9000chosen per machine or deployment3. Config fileport = 8080shared, kept with the project4. Defaults in the codeport = 8000always therenot set herenot set herenot set here

Where a setting comes from: the first layer that sets it wins

Text description of the diagram

The diagram is a column of four boxes under the question "the program asks for a setting". The program looks in each box in turn and stops at the first one that sets the value; only when a layer does not set it does the search go down to the next one.

  1. Command-line flags, such as --port=9100, chosen for one run of the program. When the flag is given, its value wins and the other layers are not asked.
  2. Environment variables, such as APP_PORT=9000, chosen for one machine or one deployment.
  3. The config file, such as port = 8080 in a TOML file, shared and kept with the project.
  4. The defaults in the code, such as port = 8000, which always exist.

collections.ChainMap looks keys up in a list of dictionaries and returns the first match, which is exactly this rule. Follow the three settings of the next program through the layers:

Interactive · stage by stage Three settings looked up through four layers Step through the lookups of precedence.py. Each setting starts at the flags and stops at the first layer that has it.
Three settings looked up through four layers, step 5 of 5. Showing 1. Command-line flags (debug: the bool False), 2. Environment variables (port: the int 9000), 3. The config file, 4. Defaults in the code.1. Command-line flagsdebug: the bool False2. Environment variablesport: the int 90003. The config fileport = 8080, debug = true4. Defaults in the code'127.0.0.1', 8000, Falselost port and debug

The drawing shows the whole scene; its 5 steps are listed below.

Every step as text
  1. Text in, typed values out. The environment and the flags hold text. convert() turns each value into the type of the setting's default before it is stored, so APP_PORT=9000 becomes the int 9000 and --debug=false the bool False.
  2. host: no layer sets it. The lookup of host asks the flags, the environment and the config file in turn. None of them has host, so ChainMap goes on to the defaults and returns '127.0.0.1'.
  3. port: the environment answers. port is not among the flags, and the environment has it, so 9000 wins. The config file's 8080 and the default 8000 are never read.
  4. debug: the flag wins. debug is a flag, so the lookup stops at the first layer with False. The config file says true, but a flag is chosen for this one run and beats every other layer.
  5. Where each value came from. The program prints each value with the layer that set it. The config file set port and debug and lost both; when a setting does not take effect, that line says which layer to look at.

Four layers in a ChainMap Python · precedence.py 
import os
import tomllib
from collections import ChainMap

defaults = {"host": "127.0.0.1", "port": 8000, "debug": False}
from_file = tomllib.loads('port = 8080\ndebug = true\n')
TRUE, FALSE = {"1", "true", "yes", "on"}, {"0", "false", "no", "off"}


def convert(key, text):
    """Text from the environment or a flag, turned into the type of the key's default."""
    if isinstance(defaults[key], bool):
        word = text.strip().lower()
        if word not in TRUE | FALSE:
            raise ValueError(f"{key} must be one of {sorted(TRUE | FALSE)}, not {text!r}")
        return word in TRUE
    return type(defaults[key])(text)


os.environ["APP_PORT"] = "9000"  # set by the shell in real life
from_env = {}
for name, text in os.environ.items():
    key = name.removeprefix("APP_").lower()
    if name.startswith("APP_") and key in defaults:
        from_env[key] = convert(key, text)

argv = ["--debug=false"]  # what sys.argv[1:] would hold
from_flags = {}
for arg in argv:
    key, _, text = arg.removeprefix("--").partition("=")
    from_flags[key] = convert(key, text)

settings = ChainMap(from_flags, from_env, from_file, defaults)  # the first map that has a key wins
layers = {"flags": from_flags, "environment": from_env, "config file": from_file, "defaults": defaults}
for key in defaults:
    source = next(name for name, layer in layers.items() if key in layer)
    print(f"{key:<5} = {settings[key]!r:<12} from the {source}")

Output

host  = '127.0.0.1'  from the defaults
port  = 9000         from the environment
debug = False        from the flags

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 precedence.py

The config file set port and debug and lost both: the environment overrode port and the flag overrode debug, while host came from the defaults because no other layer set it. Two things in the example are worth copying. convert() turns text from the environment and from flags into the type of the setting’s default before it is stored, so the program never sees "9000" where it expects 9000. And the program can say where each value came from, which saves a lot of guessing when a setting does not take effect. Press Edit, add "--port=9100" to argv and run it again: the port now comes from the flags, as in the diagram.

Key idea

Look every setting up in one fixed order, from the flags of this run to the defaults in the code, and convert text to the type of the default as it comes in.

Checkpoint

Check: Which layer wins

  1. Question 1 of 2 The defaults say port = 8000, the config file port = 8080, the environment APP_PORT=9000, and the program runs with --port=9100. With the order of the lesson, which port does it use?

    Choose one answer.

    Show the answer

    Answer: 9100, from the flag

    The lookup goes from the most specific layer to the least: flags, environment, config file, defaults. The first layer that has the key wins, and here that is the flag, 9100.

  2. Question 2 of 2 A loader stores APP_PORT=9000 in its settings as it finds it, without converting. What goes wrong later?

    Choose one answer.

    Show the answer

    Answer: The port is the str '9000', so code such as settings["port"] + 1 raises TypeError

    A value from the environment or a flag is text, while the default and the config file give an int. Convert every value to the type of its default as it comes in, so the program sees one type whichever layer won.

Secrets stay out of the code and out of other programs

A password or an API key written into the source code is shared with everyone who can read the code, now and in every clone, fork, backup and build log of it. Deleting the line in a later commit does not help, because the old commit still holds it. Treat a secret that reached version control as known to others: revoke it and issue a new one first, then clean up. To keep secrets out in the first place, programs read them at run time.

For local development, a .env file that tools load into the environment is common; list it in .gitignore, and commit a .env.example with the names of the variables and no values. Environment variables are much better than code, but they are not private. The OWASP Secrets Management Cheat Sheet notes that they are generally accessible to all processes and may end up in logs or system dumps, and it recommends a secrets manager: A service built to store credentials such as passwords and API keys and hand each program only the ones it may read, at run time, so they stay out of code, config files and the environment., or secret files that the platform mounts into the container, over plain environment variables. One reason is easy to see: every child process: A program that another program starts, for example with subprocess.run(). It receives a copy of its parent's whole environment unless the parent passes env= with only the variables it chooses. starts with a copy of the whole environment. Step through a program that starts the same helper twice:

Interactive · message by message A child process gets a copy of the whole environment Step through child_env.py. The program starts the same helper twice, once with its own environment and once with env= set.
A child process gets a copy of the whole environment, step 5 of 5. Messages so far: The shell to Your program: SHOP_API_KEY=…7f3a; Your program to A helper program: the whole environment; Your program to A helper program: env=clean, no _KEY names.The shellstarts your programYour programholds SHOP_API_KEYA helper programneeds no keySHOP_API_KEY=…7f3athe whole environmentenv=clean, no _KEY nameschild sees: None

The drawing shows the whole scene; its 5 steps are listed below.

Every step as text
  1. The key arrives. The shell starts your program with SHOP_API_KEY in its environment. The example sets the variable itself, which has the same effect on every program it starts afterwards.
  2. A helper starts. subprocess.run() starts a helper that has no use for the key. Without env=, the helper receives a copy of the whole environment of your program.
  3. The helper sees the key. The helper prints "child sees: example-only-key-7f3a". It holds the key now, and so does every program it starts in turn.
  4. Only what it needs. The second run passes env=clean, a dict without the variables whose names end in _KEY. subprocess uses that mapping instead of the inherited environment.
  5. The key stays home. The helper prints "child sees: None". Pass env= with only the variables a program needs whenever you start one that should not hold your credentials.

A child process sees the parent's secrets Python · child_env.py 
import os
import subprocess
import sys

os.environ["SHOP_API_KEY"] = "example-only-key-7f3a"  # pretend the shell set it

child = 'import os; print("child sees:", os.environ.get("SHOP_API_KEY"))'
subprocess.run([sys.executable, "-c", child])  # a child process gets a copy of the whole environment

clean = {name: value for name, value in os.environ.items() if not name.endswith("_KEY")}
subprocess.run([sys.executable, "-c", child], env=clean)  # pass only what the child needs

Output

child sees: example-only-key-7f3a
child sees: None

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 child_env.py

The first helper printed the key, the second None. Passing env= to subprocess.run() gives the child exactly the variables you choose instead of a copy of yours. Do that whenever you start a program that does not need your credentials.

Security

If a secret ever lands in version control, a log, a chat or a screenshot, revoke and replace it first and clean up afterwards: deleting the line does not remove the copies people and tools already have.

Checkpoint

Check: Where secrets come from

  1. Question 1 of 2 An API key was committed and pushed, and the next commit deleted the line. What do you do first?

    Choose one answer.

    Show the answer

    Answer: Revoke the key and issue a new one, then clean up

    Treat a secret that reached version control as known to others. Revoke it and issue a new one first, give the new one to the program at run time, and only then clean up the history and the logs.

  2. Question 2 of 2 Your program holds SHOP_API_KEY in its environment and runs subprocess.run(["convert", "in.png", "out.jpg"]). What can the converter see?

    Choose one answer.

    Show the answer

    Answer: The key, because a child process gets a copy of the whole environment

    A child process inherits the environment of its parent. Pass env= with only what the program needs, as the second run of child_env.py does, whenever you start one that should not hold your credentials.

A masking wrapper keeps secrets out of logs

Logs, traceback: The report Python prints when an exception ends a program: the calls that led to the failing line, oldest first, and on the last line the exception's type and message. More in “Handling exceptions with try and except” and error messages print values without asking. A settings dictionary logged at start-up prints every value it holds, and an exception message that repeats its input can publish a password. A small wrapper class whose repr() shows only a masked form protects every such place at once:

A Secret class and a mask() helper Python · mask_secret.py 
import logging
import sys

logging.basicConfig(stream=sys.stdout, format="%(levelname)s %(message)s", level=logging.INFO)


def mask(value):
    """At most the last four characters of a secret, and only when it is long enough to spare them."""
    if not value:
        return "(not set)"
    return "****" + value[-4:] if len(value) >= 12 else "****"


class Secret:
    """Holds a secret value; printing, logging or formatting it shows only a masked form."""

    def __init__(self, value):
        self._value = value

    def reveal(self):
        return self._value

    def __repr__(self):
        return f"Secret({mask(self._value)})"

    __str__ = __repr__


api_key = Secret("example-only-key-7f3a")  # in real code: Secret(os.environ["SHOP_API_KEY"])
settings = {"host": "api.example.com", "timeout": 10, "api_key": api_key}

logging.info("starting with %s", settings)  # the dictionary's repr() shows Secret(****7f3a)
print(f"key used: {api_key}")
print("short secret:", mask("abc123"))

# What the HTTP client needs gets the real value, and nothing else does.
headers = {"Authorization": f"Bearer {api_key.reveal()}"}
print("header length:", len(headers["Authorization"]))

Output

INFO starting with {'host': 'api.example.com', 'timeout': 10, 'api_key': Secret(****7f3a)}
key used: Secret(****7f3a)
short secret: ****
header length: 28

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 mask_secret.py

logging builds each message with the % operator, which calls str() on its arguments, and f-strings call str() too, so every place the program printed the key shows the masked form:

Where the key went What was printed
the settings dict, logged at start-up Secret(****7f3a) inside the dict
an f-string key used: Secret(****7f3a)
mask("abc123"), a short secret ****, nothing of the value
the Authorization header, through reveal() the real key: Bearer and its 21 characters, 28 in all

Only the code that really needs the value, here the HTTP header, calls reveal(). Short secrets are masked completely, because four characters of a six-character code give away too much. Error messages follow the same rule: say which setting is wrong and why, never what its value was.

A wrapper protects only the paths that use it, so check the result the way a test would: capture what the program logs and search it for the secret itself. The check fails on the plain string and passes on the wrapper:

A test that searches the log for the key Python · leak_check.py 
import io
import logging

KEY = "example-only-key-7f3a"  # in real code it comes from os.environ


class Secret:
    """Shows only its last four characters when it is printed, formatted or logged."""

    def __init__(self, value):
        self._value = value

    def reveal(self):
        return self._value

    def __repr__(self):
        return f"Secret(****{self._value[-4:]})"

    __str__ = __repr__


def start_up_log(api_key):
    """Everything a service logs at start-up, captured as text so that a test can search it."""
    stream = io.StringIO()
    log = logging.getLogger("service")
    log.propagate = False
    handler = logging.StreamHandler(stream)
    log.addHandler(handler)
    log.warning("starting with %s", {"host": "api.example.com", "api_key": api_key})
    log.warning("retrying the request with key %s", api_key)
    log.removeHandler(handler)
    return stream.getvalue()


# The check a test suite can keep: search the log for the secret itself.
for label, api_key in [("plain str", KEY), ("Secret", Secret(KEY))]:
    text = start_up_log(api_key)
    print(f"{label}: the key is in the log: {KEY in text}")
    for line in text.splitlines():
        print("  ", line)

Output

plain str: the key is in the log: True
   starting with {'host': 'api.example.com', 'api_key': 'example-only-key-7f3a'}
   retrying the request with key example-only-key-7f3a
Secret: the key is in the log: False
   starting with {'host': 'api.example.com', 'api_key': Secret(****7f3a)}
   retrying the request with key Secret(****7f3a)

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 leak_check.py

The plain string put the key into both log lines and the wrapper into neither, so the same check, kept in a test suite, fails the moment any code path prints the key.

Logging settings, or a value in an error message

logging.info("settings: %s", settings) and raise ValueError(f"bad key {key!r}") both print the secret. Wrap secrets in a class like Secret, and build error messages from the setting’s name and the rule it broke.

Checkpoint

Check: Secrets in logs and errors

  1. Question 1 of 2 With the lesson's mask(), which shows the last four characters only of secrets of 12 characters or more, what is mask("pin-4471")?

    Choose one answer.

    Show the answer

    Answer: '****'

    mask() returns "(not set)" for an empty value, "**" plus the last four characters for 12 characters or more, and "**" alone for anything shorter, so a short code gives nothing away.

  2. Question 2 of 2 You want a test that fails whenever start-up logging leaks the API key. Which check does that?

    Choose one answer.

    Show the answer

    Answer: Capture what the program logs and assert that the key itself does not appear in it

    leak_check.py captures the log in a StringIO and checks KEY in text: True for the plain string, False for the Secret wrapper. The same check in a test fails the moment any code path prints the key.

Secret & Credential Leak Scanner Paste a config file, a .env file or a log to look for API keys and passwords before you commit or share it.

Practise: load settings from four layers

The exercise asks for load_settings(), which merges the defaults, a TOML text, the APP_ variables and the flags with the last layer winning, converts and checks every value, and refuses a missing or short API key without ever putting the key in an error message, and for the mask() helper. Its sample tests run in your browser:

Exercise · Medium · Python

Merge settings from four layers and keep the key secret

settings.py defines DEFAULTS = {"host": "127.0.0.1", "port": 8000, "debug": False, "api_key": None}. Write two functions.

  • mask(value) returns "(not set)" for None or "", "****" followed by the last four characters for a value of 12 characters or more, and "****" alone for anything shorter.
  • load_settings(toml_text, environ, argv) returns a new dictionary with the four settings, merged from four layers, where a later layer wins: DEFAULTS, then the TOML text, then the environment, then the command-line flags.
    • TOML keys are the setting names. A key that is not a setting raises ValueError, and so does a value whose type is not the type of the default (api_key takes a string).
    • environ is a dictionary such as os.environ. Only the variables APP_HOST, APP_PORT, APP_DEBUG and APP_API_KEY count; ignore every other variable.
    • argv is a list such as ["--port=9000", "--debug=yes"]. A flag that is not --<setting>=<value> for one of the four settings raises ValueError.
    • Values from the environment and from flags are text. Convert them to the type of the default: int() for the port, and for debug 1, true, yes or on mean True and 0, false, no or off mean False, in any mix of upper and lower case. Any other text raises a ValueError that names the variable or the flag.
    • The API key is required. Raise ValueError when it is missing or empty after merging, with a message that says to set APP_API_KEY, and when it is shorter than 16 characters. No error message may contain the key itself: use mask() if you want to show it.

Starter code · settings.py 

import tomllib

DEFAULTS = {"host": "127.0.0.1", "port": 8000, "debug": False, "api_key": None}


def mask(value):
    """A form of a secret that is safe to print."""
    # Replace this line with your code.
    return value


def load_settings(toml_text, environ, argv):
    """DEFAULTS, then the TOML text, then APP_ variables, then --name=value flags; the last layer wins."""
    # Replace these lines with your code.
    settings = dict(DEFAULTS)
    settings.update(tomllib.loads(toml_text))
    return settings
The sample tests · test_settings.py 
from settings import load_settings, mask

KEY = "example-only-key-0123456789"


def error_of(*args):
    """The ValueError load_settings raises for these arguments (fails the test if there is none)."""
    try:
        load_settings(*args)
    except ValueError as err:
        return str(err)
    raise AssertionError(f"load_settings{args!r} should raise ValueError")


def test_mask():
    """shows at most the last four characters, and only of long values"""
    assert mask(KEY) == "****6789"
    assert mask("abc123") == "****"
    assert mask(None) == "(not set)"
    assert mask("") == "(not set)"


def test_defaults_and_key_from_environment():
    """starts from the defaults and reads the key from APP_API_KEY"""
    settings = load_settings("", {"APP_API_KEY": KEY, "PATH": "/usr/bin", "HOME": "/home/asha"}, [])
    assert settings == {"host": "127.0.0.1", "port": 8000, "debug": False, "api_key": KEY}


def test_each_layer_wins_over_the_one_before():
    """flags beat the environment, which beats TOML, which beats the defaults"""
    toml_text = 'host = "0.0.0.0"\nport = 8080\ndebug = true\n'
    environ = {"APP_PORT": "9000", "APP_DEBUG": "Off", "APP_API_KEY": KEY}
    settings = load_settings(toml_text, environ, ["--debug=YES"])
    assert settings == {"host": "0.0.0.0", "port": 9000, "debug": True, "api_key": KEY}
    assert load_settings(toml_text, environ, [])["debug"] is False
    assert load_settings("", {"APP_API_KEY": KEY}, ["--debug=1", "--host=example.org"])["host"] == "example.org"


def test_bad_values_name_their_source():
    """refuses text that is not a number or a yes/no word, naming the variable or the flag"""
    assert "APP_PORT" in error_of("", {"APP_PORT": "eighty", "APP_API_KEY": KEY}, [])
    assert "APP_DEBUG" in error_of("", {"APP_DEBUG": "maybe", "APP_API_KEY": KEY}, [])
    assert "--port" in error_of("", {"APP_API_KEY": KEY}, ["--port=80.5"])


def test_unknown_names_and_wrong_types():
    """refuses unknown TOML keys and flags, and TOML values of the wrong type"""
    error_of('colour = "teal"\n', {"APP_API_KEY": KEY}, [])
    error_of('port = "8080"\n', {"APP_API_KEY": KEY}, [])
    error_of("", {"APP_API_KEY": KEY}, ["--verbose=true"])
    error_of("", {"APP_API_KEY": KEY}, ["port=80"])


def test_api_key_is_required_and_never_shown():
    """asks for APP_API_KEY when the key is missing, and never puts a key in a message"""
    assert "APP_API_KEY" in error_of("", {}, [])
    assert "APP_API_KEY" in error_of("", {"APP_API_KEY": ""}, [])
    message = error_of('api_key = "short-key-91"\n', {}, [])
    assert "short-key-91" not in message
    message = error_of("", {"APP_API_KEY": KEY}, ["--port=eighty"])
    assert KEY not in message
A hint

Start from settings = dict(DEFAULTS) and apply the layers in order with one helper that converts and checks a value for a given setting name, so the TOML text, the environment and the flags share the same rules. tomllib.loads() reads the TOML text; for a flag, arg.removeprefix("--").partition("=") splits it into a name and a value. Check the API key only after the last layer, and build its error message without the key.

The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.

The practice below covers TOML, INI files, the environment, the order of the layers and both halves of keeping secrets. It starts with the part you have answered least well, and every answer comes back with the reason it holds or fails:

Practice that adapts to you

Practise this lesson

13 questions about 6 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.

  1. Question 1 of 13 Which call reads a TOML file correctly?

    Choose one answer.

    Show the answer

    Answer: with open("app.toml", "rb") as f: data = tomllib.load(f)

    Open the file in binary mode and pass it to tomllib.load(), or read the text yourself and pass the string to tomllib.loads().

  2. Question 2 of 13 What does this print?

    Read the code, then choose one answer.

    import tomllib

settings = tomllib.loads('port = "8080"\n')
try:
    print(settings["port"] + 1)
except TypeError as err:
    print("TypeError:", err)
    Show the answer

    Answer: TypeError: can only concatenate str (not "int") to str

    In TOML the way a value is written decides its type: port = 8080 is an integer and port = "8080" a string. tomllib returns exactly those types.

  3. Question 3 of 13 Which values does getboolean() read as True?

    Choose every answer that is right.

    Show the answer

    Answer:

    • On
    • yes

    getboolean() accepts 1, yes, true and on for True and 0, no, false and off for False, in any case, and raises ValueError for anything else.

  4. Question 4 of 13 A program calls config.read(["defaults.ini", "local.ini"]), and defaults.ini is missing because of a typo in the install. What happens, and what should the program do?

    Choose one answer.

    Show the answer

    Answer: read() skips it and returns ['local.ini']; check the list, or read the required file with read_file(open(...))

    config.read() returns the names of the files it parsed and silently skips the rest. For a file the program cannot run without, open it yourself and pass it to read_file(), so a missing file raises an error.

  5. Question 5 of 13 What type is every value in os.environ?

    Choose one answer.

    Show the answer

    Answer: str

    os.environ maps names to strings. Convert numbers with int() and switches by comparing the text with the words you accept.

  6. Question 6 of 13 The shell started the program with DEBUG=false. What does this print?

    Read the code, then choose one answer.

    import os

debug = bool(os.environ.get("DEBUG"))
print("debug is on" if debug else "debug is off")
    Show the answer

    Answer: debug is on

    os.environ.get("DEBUG") returns the string "false", and bool() of any non-empty string is True. Compare the text with the words you accept, such as {"1", "true", "yes", "on"}, instead.

  7. Question 7 of 13 APP_WORKERS=four reaches a service whose code is int(os.environ["APP_WORKERS"]). Which error message helps the person who set it most?

    Choose one answer.

    Show the answer

    Answer: ValueError: APP_WORKERS must be a whole number, not 'four'

    Read the text, convert it, and raise a ValueError that names the variable and the rule, as env_int() does. For a secret, name the variable and the rule only, never the value.

  8. Question 8 of 13 Put these sources of a setting in the usual order, from the one that loses to every other to the one that wins over every other.

    Give each item its position, from 1 (first).

    Show the answer

    Answer:

    1. Defaults in the code
    2. The config file
    3. Environment variables
    4. Command-line flags

    Each layer is more specific than the one before it: defaults hold for every installation, the config file for a project, environment variables for one machine or deployment, and flags for one run, so the most specific value wins.

  9. Question 9 of 13 What does precedence.py print?

    What does this program print? Choose one answer.

    import os
import tomllib
from collections import ChainMap

defaults = {"host": "127.0.0.1", "port": 8000, "debug": False}
from_file = tomllib.loads('port = 8080\ndebug = true\n')
TRUE, FALSE = {"1", "true", "yes", "on"}, {"0", "false", "no", "off"}


def convert(key, text):
    """Text from the environment or a flag, turned into the type of the key's default."""
    if isinstance(defaults[key], bool):
        word = text.strip().lower()
        if word not in TRUE | FALSE:
            raise ValueError(f"{key} must be one of {sorted(TRUE | FALSE)}, not {text!r}")
        return word in TRUE
    return type(defaults[key])(text)


os.environ["APP_PORT"] = "9000"  # set by the shell in real life
from_env = {}
for name, text in os.environ.items():
    key = name.removeprefix("APP_").lower()
    if name.startswith("APP_") and key in defaults:
        from_env[key] = convert(key, text)

argv = ["--debug=false"]  # what sys.argv[1:] would hold
from_flags = {}
for arg in argv:
    key, _, text = arg.removeprefix("--").partition("=")
    from_flags[key] = convert(key, text)

settings = ChainMap(from_flags, from_env, from_file, defaults)  # the first map that has a key wins
layers = {"flags": from_flags, "environment": from_env, "config file": from_file, "defaults": defaults}
for key in defaults:
    source = next(name for name, layer in layers.items() if key in layer)
    print(f"{key:<5} = {settings[key]!r:<12} from the {source}")
    Show the answer

    Answer: it prints 

    host  = '127.0.0.1'  from the defaults
port  = 9000         from the environment
debug = False        from the flags

    ChainMap returns the value of the first dictionary that has the key: the flags set debug, the environment set port, and only the defaults set host. The example converts the environment's text and the flag's text before storing them, so port is the int 9000 and debug the bool False.

  10. Question 10 of 13 Which of these keep an API key out of places where others can read it?

    Choose every answer that is right.

    Show the answer

    Answer:

    • Starting helper programs with subprocess.run(..., env=only_what_they_need)
    • Reading it at run time from a secrets manager or a mounted secret file
    • Wrapping it in a class whose repr() shows only a masked form

    A committed key stays in the history even after the line is removed, and a logged settings dictionary prints every value it holds. A secrets manager, a masking wrapper and an explicit env= for child processes each keep the key away from people and programs that do not need it.

  11. Question 11 of 13 A project loads its local settings from a .env file. What belongs in version control?

    Choose one answer.

    Show the answer

    Answer: A .env.example with the names of the variables and no values, with .env listed in .gitignore

    Commit the names in a .env.example and keep the values out: list .env in .gitignore, and in production read secrets from a secrets manager or a mounted file.

  12. Question 12 of 13 What does mask_secret.py print?

    What does this program print? Choose one answer.

    import logging
import sys

logging.basicConfig(stream=sys.stdout, format="%(levelname)s %(message)s", level=logging.INFO)


def mask(value):
    """At most the last four characters of a secret, and only when it is long enough to spare them."""
    if not value:
        return "(not set)"
    return "****" + value[-4:] if len(value) >= 12 else "****"


class Secret:
    """Holds a secret value; printing, logging or formatting it shows only a masked form."""

    def __init__(self, value):
        self._value = value

    def reveal(self):
        return self._value

    def __repr__(self):
        return f"Secret({mask(self._value)})"

    __str__ = __repr__


api_key = Secret("example-only-key-7f3a")  # in real code: Secret(os.environ["SHOP_API_KEY"])
settings = {"host": "api.example.com", "timeout": 10, "api_key": api_key}

logging.info("starting with %s", settings)  # the dictionary's repr() shows Secret(****7f3a)
print(f"key used: {api_key}")
print("short secret:", mask("abc123"))

# What the HTTP client needs gets the real value, and nothing else does.
headers = {"Authorization": f"Bearer {api_key.reveal()}"}
print("header length:", len(headers["Authorization"]))
    Show the answer

    Answer: it prints 

    INFO starting with {'host': 'api.example.com', 'timeout': 10, 'api_key': Secret(****7f3a)}
key used: Secret(****7f3a)
short secret: ****
header length: 28

    Every place that prints the Secret goes through repr() or str(), which show Secret(****7f3a). mask() hides short secrets completely, and the header gets the real value through reveal(): Bearer and a space plus the 21-character key make 28 characters.

  13. Question 13 of 13 A loader refuses an API key that is too short. Which error message is safe to log?

    Choose one answer.

    Show the answer

    Answer: APP_API_KEY must have at least 16 characters

    Error messages say which setting is wrong and why, never what its value was. If a message must show a secret, use the masked form, which hides short secrets completely.

Interview questions

Why is every value in os.environ a string, and how should you read a boolean setting such as DEBUG from it?

The operating system stores the environment of a process as text, pairs such as DEBUG=false, and it passes the same text to every program, whatever language it is written in. Python cannot know which values were meant as numbers or switches, so os.environ gives you strings and the conversion is yours. For a switch, bool(os.environ["DEBUG"]) is wrong, because every non-empty string is true, "false" and "0" included. Read the text with os.environ.get("DEBUG", ""), normalise it with strip().lower(), compare it with explicit sets such as {"1", "true", "yes", "on"} and {"0", "false", "no", "off", ""}, and raise a ValueError that names the variable for anything else, so a typo cannot silently switch debugging on or off.

Key takeaways

  • tomllib.loads() and tomllib.load(), with a file opened "rb", give typed values; writing TOML needs a package such as tomli-w or tomlkit.
  • configparser returns strings, inherits [DEFAULT] into every section, ignores the case of keys and treats % as interpolation unless you pass interpolation=None.
  • os.environ holds text only, so bool("false") is True: compare switches with the words you accept and name the variable in every error.
  • Look settings up in one fixed order, flags, environment, config file, defaults, and convert text to the type of the default as it comes in.
  • Secrets never go into code or version control: a leaked one is revoked and replaced first, production keeps them in a secrets manager or mounted files, and child processes get only the variables they need.
  • Print secrets only in masked form, through a wrapper whose repr() hides them, and test your logs for the secret itself.

Terms in this lesson

child process
A program that another program starts, for example with subprocess.run(). It receives a copy of its parent's whole environment unless the parent passes env= with only the variables it chooses.
environment variable
A name with a text value that a process receives from the program that started it, such as APP_PORT=9000. Python reads them from os.environ, always as str, and every process it starts gets a copy of them.
interpolation (configparser)
Replacing a reference such as %(host)s inside an INI value with another value of the same section or of [DEFAULT], when the value is read. A lone % raises InterpolationSyntaxError; interpolation=None turns it off.
secrets manager
A service built to store credentials such as passwords and API keys and hand each program only the ones it may read, at run time, so they stay out of code, config files and the environment.

References

Related tools

TOML ↔ JSON Converter Exact TOML ⇄ JSON / YAML conversion, with errors that name the rule and the line. Secret & Credential Leak Scanner Find leaked keys, tokens and passwords in code and logs before someone else does.

Report a problem with this lesson