List, dict and set comprehensions

A loop that keeps the passing marks of a class takes four lines: an empty list, a for, an if and an append(). A comprehensioncomprehension: An expression in square brackets or braces that builds a new list, dict or set from the items of an iterable, such as [m for m in marks if m >= 40]. It runs a loop of its own and never changes what it reads. does the same job in one expression, [m for m in marks if m >= 40], which reads almost like the sentence it stands for: each mark, for every mark in marks, if that mark is at least 40. It never changes the list it reads, and there is a form for each built-in container: square brackets for a list, braces with key: value for a dictionary, and braces with a single expression for a set.

Below you can run seven marks through a comprehension one at a time, watch a dict comprehension lose a student, step through the two loops that two for clauses stand for, see which names a comprehension can reach, and use up a generator expression. The last section shows where a comprehension stops being clearer than the loop it replaces.

A comprehension is a loop that builds a new list

Here is a list of marks filtered twice, once with a loop and once with a comprehension, followed by a comprehension that changes every item and one that does both:

marks = [72, 35, 88, 41, 29, 95, 60]

# The loop: start empty, test each mark, append the ones that pass.
passed_loop = []
for m in marks:
    if m >= 40:
        passed_loop.append(m)

# The comprehension: the same three parts in one expression.
passed = [m for m in marks if m >= 40]

print(passed_loop)
print(passed)
print(passed == passed_loop)

# Without an if clause, every item makes it into the new list.
with_grace = [m + 5 for m in marks]
print(with_grace)

# Both clauses: keep the passing marks, then add 5 to each of them.
print([m + 5 for m in marks if m >= 40])
print(marks)  # the original list is unchanged

[72, 88, 41, 95, 60]
[72, 88, 41, 95, 60]
True
[77, 40, 93, 46, 34, 100, 65]
[77, 93, 46, 100, 65]
[72, 35, 88, 41, 29, 95, 60]

The comprehension has the same three parts as the loop, in a different order:

• The for clause, for m in marks, is the header of the loop without its colon. It names each item in turn. Any iterableiterable: A value a for loop can go through one item at a time, such as a string, a list, a tuple, a dict, a set or a range. A dict gives its keys. More in “for loops with range, enumerate and zip” works here: a list, a range(), a string, a dictionary or the lines of a file.
• The if clause, if m >= 40, is optional. An item is added only when the condition is true.
• The expression before for, here plain m, says what to put in the new list for each item that was kept. In [m + 5 for m in marks] it is m + 5, and without an if clause every item is used.

The expression is written first, so a comprehension reads as “what you get, then where it comes from”, but it runs last for each item. Step through the comprehension that does both jobs, [m + 5 for m in marks if m >= 40], on the same seven marks:

35 and 29 never reached the expression, and the other five came out 5 marks higher, in their original order: [77, 93, 46, 100, 65], the fifth line of the output. The last line matters as much as the others: marks still holds the original numbers. A comprehension always builds a new list, which you keep by assigning it to a name.

if after for filters; if-else before for picks a value

An if can appear in two places, and the place decides what it does. After the for clause it filters: items that fail the test are left out, so the new list can be shorter. Before the for clause, as a if condition else b, it is a conditional expressionconditional expression: An expression of the form a if condition else b. It evaluates the condition and gives a when the condition is true and b otherwise, evaluating only the side it picks. More in “Booleans, None, truthiness and comparisons” that chooses a value for every item, so the new list always has the same length as the input. The readings below come from a sensor that writes -999 when a measurement fails:

readings = [21.5, -999, 23.0, 22.4, -999, 24.1]  # -999 marks a failed reading

# if after the for clause filters: failed readings are left out.
valid = [r for r in readings if r != -999]

# if ... else before the for clause chooses a value: every reading stays.
filled = [r if r != -999 else None for r in readings]

print(len(readings), "readings")
print(len(valid), "valid:", valid)
print(len(filled), "filled:", filled)
print("average of the valid ones:", round(sum(valid) / len(valid), 2))

6 readings
4 valid: [21.5, 23.0, 22.4, 24.1]
6 filled: [21.5, None, 23.0, 22.4, None, 24.1]
average of the valid ones: 22.75

Use the filter when the bad items should disappear, as for the average, which must not count -999 as a temperature. Use the conditional expression when positions matter, for example when reading number 2 must stay reading number 2 in a chart, with None in place of the failed value. Press Edit, change the filter’s test to r > 22 and run it: three readings are left, and the average of the valid ones becomes 23.17.

The table sets the two forms side by side, with the six readings of the program:

Mixing the two forms up gives a syntax error. An else after the for clause is not part of any expression, so Python stops before it runs a single line:

readings = [21.5, -999, 23.0]

# compile() turns text into code without running it, so this program can
# catch the SyntaxError and print it instead of stopping.
wrong = "[r for r in readings if r != -999 else 0]"
try:
    compile(wrong, "<attempt>", "eval")
except SyntaxError as err:
    print(f"{wrong}\nSyntaxError: {err.msg}")

# The else belongs to a conditional expression, which comes before "for":
print([r if r != -999 else 0 for r in readings])

[r for r in readings if r != -999 else 0]
SyntaxError: invalid syntax
[21.5, 0, 23.0]

Python 3.14 reports only invalid syntax, and when the line is in a script, the traceback’s markers point at the else. The fix is to move the whole if ... else in front of for.

Braces make dict and set comprehensions

Braces make the other two kinds. A dict comprehension has a key: value pair where the list form has its expression, and a set comprehension has a single expression:

marks = {"Asha": 91, "Ravi": 67, "Meera": 78, "John": 45, "Divya": 88}


def grade(mark):
    if mark >= 85:
        return "A"
    if mark >= 70:
        return "B"
    if mark >= 50:
        return "C"
    return "D"


# Dict comprehension: a key: value pair for every item.
grades = {name: grade(mark) for name, mark in marks.items()}
print(grades)

# Set comprehension: each grade once, however many students have it.
given = {g for g in grades.values()}
print(sorted(given))

# Swapping keys and values: a key can appear only once,
# so when two students share a grade, the later one wins.
by_grade = {g: name for name, g in grades.items()}
print(by_grade)

# To keep everyone, build a list for each grade instead.
everyone = {g: [n for n, x in grades.items() if x == g] for g in "ABCD"}
print(everyone)

{'Asha': 'A', 'Ravi': 'C', 'Meera': 'B', 'John': 'D', 'Divya': 'A'}
['A', 'B', 'C', 'D']
{'A': 'Divya', 'C': 'Ravi', 'B': 'Meera', 'D': 'John'}
{'A': ['Asha', 'Divya'], 'B': ['Meera'], 'C': ['Ravi'], 'D': ['John']}

• {name: grade(mark) for name, mark in marks.items()} builds the grade book. The for clause unpacks each pair from items() into two names, exactly as a for loop over items() does.
• {g for g in grades.values()} keeps each grade once: a set holds no duplicates. Sets have no fixed order, so the program sorts the grades before printing them, which keeps the output the same from run to run.

Swapping keys and values loses data whenever two keys share a value. A dictionary holds each key once, and when a comprehension produces the same key again, the later value replaces the earlier one. Step through by_grade to see who is lost:

Asha’s "A" was replaced by Divya’s, and nothing warned about it. To keep everyone, make the value a list, as the last comprehension of the program does with a comprehension inside it.

Empty braces {} are an empty dictionary, never an empty set; set() makes an empty set. A set comprehension with no items to add, such as {x for x in []}, also gives set().

Two for clauses nest like loops; an inner comprehension builds rows

A comprehension may have several for clauses, and if clauses between them. They nest from left to right, just as the loops would be written one inside the other, and the result is one flat list. Step through the two loops that [m for row in marks for m in row] stands for, on the first two rows of a table of marks:

marks = [[78, 64, 90], [55, 81, 70]]
flat = []
for row in marks:
    for m in row:
        flat.append(m)
print(flat)

[78, 64, 90, 55, 81, 70]

The inner loop read all three marks of a row before the outer loop moved on to the next row, so flat holds the marks row by row. A comprehension can also sit inside the expression of another one, and then each inner comprehension builds a list of its own. The two look alike and do different things, as the full table shows, with one row per student and one column per subject:

# One row per student, one column per subject: maths, physics, chemistry.
marks = [
    [78, 64, 90],
    [55, 81, 70],
    [92, 88, 61],
    [40, 59, 75],
]

# Two for clauses, read left to right like two nested loops: one flat list.
flat = [m for row in marks for m in row]
print(flat)

# The same order with the loops written out:
flat_loop = []
for row in marks:
    for m in row:
        flat_loop.append(m)
print(flat == flat_loop)

# A comprehension inside the expression: one new list per column.
by_subject = [[row[col] for row in marks] for col in range(len(marks[0]))]
for subject, column in zip(["maths", "physics", "chemistry"], by_subject):
    print(f"{subject:9}", column, "best:", max(column))

# zip(*marks) pairs up the columns too, as tuples.
print(list(zip(*marks)))

[78, 64, 90, 55, 81, 70, 92, 88, 61, 40, 59, 75]
True
maths     [78, 55, 92, 40] best: 92
physics   [64, 81, 88, 59] best: 88
chemistry [90, 70, 61, 75] best: 90
[(78, 55, 92, 40), (64, 81, 88, 59), (90, 70, 61, 75)]

• [m for row in marks for m in row] has two for clauses. Read them in the order they are written: for each row, for each mark in that row. The loop version below it produces the same list, which is a good way to check any comprehension you find hard to read.
• [[row[col] for row in marks] for col in range(len(marks[0]))] has a comprehension as its expression. The outer one runs once per column, and each time the inner one collects that column from every row, so the result is a list of three lists.
• zip(*marks) does the same transposition with a built-in: the * passes each row as a separate argument, and zip() pairs up their first items, then their second, and so on, giving tuples.

import sys

grid = [[1, 2], [3], [4, 5]]
source = "[*row for row in grid]"  # unpacking inside a comprehension (PEP 798)

print(f"Python {sys.version_info.major}.{sys.version_info.minor}")
try:
    print(eval(source))
except SyntaxError as err:
    print("SyntaxError:", err.msg)

# The two-clause form gives the same list on every version:
print([x for row in grid for x in row])

Python 3.14
SyntaxError: iterable unpacking cannot be used in comprehension
[1, 2, 3, 4, 5]

A comprehension’s own variables stay inside it

A comprehension runs in a scopescope: The region of a program in which a name is looked up. Python has four kinds: the local scope of a function, enclosing functions, the global scope of a module and the built-ins. More in “Scope: local, enclosing, global and built-in” of its own, so the names its for clauses assign never leak into the code around it. Two exceptions are worth knowing, and this program shows the rule and both of them:

x = "kept"
squares = [x * x for x in range(4)]
print(squares, x)  # the comprehension's x did not replace this one

# An assignment expression (:=) is the exception: it binds in the
# enclosing scope, so total is still there after the comprehension.
total = 0
running = [total := total + n for n in [5, 3, 8]]
print(running, total)


class Exam:
    max_marks = 50
    raw = [20, 35, 50]
    # The first iterable is evaluated in the class body, so raw is found.
    copied = [m for m in raw]
    # Everything else runs in the comprehension's own scope, which cannot
    # see the names of a class body.
    try:
        percent = [m * 100 // max_marks for m in raw]
    except NameError as err:
        print("NameError:", err)


print(Exam.copied)

[0, 1, 4, 9] kept
[5, 8, 16] 16
NameError: name 'max_marks' is not defined
[20, 35, 50]

Step through the names of the program as its four comprehensions run. The boxes on the left belong to the module or to the class body, the round ones on the right to the comprehensions themselves:

• The loop variable is private. The comprehension’s x went through 0 to 3, and the outer x still holds "kept" afterwards.
• An assignment expressionassignment expression: name := value, nicknamed the walrus. It binds the value to the name and also gives the value back, so it can sit inside a larger expression such as a while condition. More in “Operators, precedence and the walrus” is not. total := total + n binds total in the enclosing scope (PEP 572 defines it that way on purpose), so a comprehension can keep a running total and leave the final value behind. Use it sparingly: a reader expects a comprehension to have no effect besides the value it returns.
• A class body is not an enclosing scope. The iterable of the first for clause is evaluated where the comprehension is written, which is why [m for m in raw] found raw. Everything else runs in the comprehension’s own scope, and that scope cannot see names defined in a class body, so max_marks is a NameError. Compute such a list after the class, where you can write Exam.max_marks, or in a function that takes the value as an argument.

Before Python 3.12, CPython ran every comprehension as a hidden function call. PEP 709 inlined list, dict and set comprehensions into the code around them, which made them faster (up to twice as fast on the PEP’s own microbenchmark) without changing what they can see: the variables stay as isolated as before. Two details did change: a traceback no longer shows a separate <listcomp> line for an error inside a comprehension, and locals() called inside one now includes the variables of the function around it.

Round brackets make a generator expression, which builds no list

Round brackets do not make a “tuple comprehension”. They make a generator expressiongenerator expression: A comprehension in round brackets, such as (x * x for x in xs). It builds no list: it makes each item only when asked for the next one, and it can be read through only once., which produces its items one at a time, only when something asks for the next one, and never builds a list at all. When it is the only argument of a call, its own brackets can be dropped: sum(x * x for x in xs). Step through a small one as it is asked for its items:

squares = (n * n for n in range(4))
print(next(squares))
print(next(squares))
rest = sum(squares)
print(rest)
print(list(squares))

0
1
13
[]

The generator handed over 0 and then 1, one call of next() each; sum() took the two squares that were left, 4 + 9 = 13; and list() found nothing at all. What a generator saves is memory, which this program measures for 1,000 and for 1,000,000 squares:

import sys

# getsizeof counts the container itself: for a list, its array of references
# (8 bytes each on a 64-bit build), not the int objects they point to.
for n in (1_000, 1_000_000):
    squares_list = [i * i for i in range(n)]  # every square, built now
    squares_gen = (i * i for i in range(n))  # a recipe: squares made one at a time
    print(f"{n:>9,} squares  list: {sys.getsizeof(squares_list):>9,} bytes  generator: {sys.getsizeof(squares_gen)} bytes")

# sum() takes the generator expression directly, without extra brackets.
print(sum(i * i for i in range(n)) == sum(squares_list))

# A generator runs once: a second pass finds it empty.
print(sum(squares_gen) > 0, sum(squares_gen))

    1,000 squares  list:     8,856 bytes  generator: 208 bytes
1,000,000 squares  list: 8,448,728 bytes  generator: 208 bytes
True
True 0

The list grew from 8,856 bytes to 8,448,728 bytes, about a thousand times more for a thousand times more squares, while the generator stayed at 208 bytes. The list object alone holds a million references, about 8 MB on a 64-bit build, before counting the integers they point to; sys.getsizeof() measures only the container. The generator is a couple of hundred bytes, whatever its length, because it keeps only its position. This example shows its output from a recorded run rather than a Run button: in the browser, Python is a 32-bit build, so every size would be different.

Two rules follow. When the values are used once, by sum(), max(), any(), all() or a for loop, a generator expression saves the memory of the list. When you need the values twice, their number or an item by position, build the list: the last line of the program shows that a generator, once used, is empty.

A loop reads better once a comprehension needs statements

A comprehension is at its best when it fits on one line and does one thing: transform, filter, or both. Beyond that it hides what a loop would show, and a loop can hold statements that a comprehension cannot, such as try and except:

lines = ["asha: 91", "", "# absent today", "ravi: 67", "meera: 104", "john: n/a"]

# One expression, three clauses and a nested conditional: it works,
# but every reader has to take it apart first, and "john: n/a" would
# crash it if the isdigit() test were forgotten.
clever = {
    name.strip().title(): min(int(score), 100) if score.strip().isdigit() else None
    for line in lines
    if line and not line.startswith("#")
    for name, score in [line.split(":")]
}
print(clever)

# The same rules as a loop: one step per line, room for a comment
# and for the error handling that a comprehension cannot hold.
plain = {}
for line in lines:
    if not line or line.startswith("#"):
        continue  # blank lines and comments carry no marks
    name, score = line.split(":")
    try:
        plain[name.strip().title()] = min(int(score), 100)  # 100 is the maximum
    except ValueError:
        plain[name.strip().title()] = None  # not a number
print(plain == clever)

# A comprehension used only for its side effect builds a useless list.
shown = [print(name) for name in plain]
print(shown)

{'Asha': 91, 'Ravi': 67, 'Meera': 100, 'John': None}
True
Asha
Ravi
Meera
John
[None, None, None, None]

The comprehension at the top is correct, and it is still a poor choice. It needs three clauses, a nested conditional and the trick of looping over a one-item list, for name, score in [line.split(":")], just to give a value a name. The loop gives every rule its own line, and it handles a score such as "n/a" with try and except, which no comprehension can contain. The last lines show the other misuse: a comprehension written only for its side effectA change a piece of code makes besides the value it returns, such as printing, writing a file or appending to another list., here printing, builds a list of None values that nobody needs. Write a for loop when nothing should be returned.

A few habits keep comprehensions readable:

• At most two clauses, say one for and one if, or two for clauses. More than that reads better as a loop.
• No side effects: no printing, no appending to another list, no changes to the items.
• A helper function for any logic that needs more than one condition, like grade() above.
• Speed is rarely the reason to choose. Comprehensions avoid looking up and calling append for every item, but the work done per item usually costs more than the loop around it, so choose the form that reads best.

Practise: word lengths and a flat list

The exercise asks for one dict comprehension and one list comprehension: a map from every cleaned word of a text to its length, and the items of a table flattened row by row. One of its sample tests reads your code to make sure the comprehensions are there, and the first hint suggests cleaning the words before deciding which to keep:

import string


def word_lengths(text):
    """Map each word of text, case-folded and without punctuation at its ends, to its length."""
    # Replace this line with a dictionary comprehension.
    return {}


def flatten_matrix(matrix):
    """Return the items of a list of lists as one flat list, row by row."""
    # Replace this line with a list comprehension.
    return []

import ast

import words
from words import flatten_matrix, word_lengths


def test_simple_sentence():
    """maps each word to its length"""
    assert word_lengths("sun rises early") == {"sun": 3, "rises": 5, "early": 5}


def test_case_and_punctuation():
    """folds case and removes punctuation at both ends of a word"""
    assert word_lengths("The cat, the HAT! (the end)") == {"the": 3, "cat": 3, "hat": 3, "end": 3}
    assert word_lengths("Straße STRASSE") == {"strasse": 7}


def test_inner_punctuation_and_empty_words():
    """keeps punctuation inside a word and leaves out words that are only punctuation"""
    assert word_lengths("don't -- stop... e-mail") == {"don't": 5, "stop": 4, "e-mail": 6}
    assert word_lengths("") == {}
    assert word_lengths("  ... !!! ") == {}


def test_flatten():
    """flattens rows in order, including empty rows"""
    assert flatten_matrix([[1, 2], [3], [], [4, 5, 6]]) == [1, 2, 3, 4, 5, 6]
    assert flatten_matrix([]) == []
    assert flatten_matrix([[], []]) == []
    grid = [["a"], ["b", "c"]]
    assert flatten_matrix(grid) == ["a", "b", "c"]
    assert grid == [["a"], ["b", "c"]], "flatten_matrix must not change its argument"


def _function(tree, name):
    for node in tree.body:
        if isinstance(node, ast.FunctionDef) and node.name == name:
            return node
    raise AssertionError(f"words.py has no function {name}()")


def test_uses_comprehensions():
    """builds the results with a dict comprehension and a list comprehension"""
    with open(words.__file__, encoding="utf-8") as source:
        tree = ast.parse(source.read())
    lengths = _function(tree, "word_lengths")
    flatten = _function(tree, "flatten_matrix")
    assert any(isinstance(n, ast.DictComp) for n in ast.walk(lengths)), "word_lengths() needs a dict comprehension"
    assert any(isinstance(n, ast.ListComp) for n in ast.walk(flatten)), "flatten_matrix() needs a list comprehension"

Seven ideas, from the parts of a comprehension to the point where a loop takes over, are in the self-check below. It starts with the one your answers mark as the weakest, and every choice carries its own reason:

Interview questions

Fresher · What is the difference between [x for x in xs if x > 0] and [x if x > 0 else 0 for x in xs]?

The first one filters: the if after the for clause decides whether each item is included, so negative numbers are left out and the result can be shorter than xs. The second one has no filter. Its x if x > 0 else 0 is a conditional expression, the part that computes each item, so every item of xs produces one item of the result and the length stays the same: negative numbers become 0. A good answer adds that the two can be combined, as in [x if x < 100 else 100 for x in xs if x > 0], and that writing else after the for clause is a syntax error, because only a filter can go there.

Key takeaways

• [expression for item in iterable if condition] builds a new list, {k: v for ...} a dictionary and {expression for ...} a set, and the iterable they read is never changed.
• An if after for filters items out, while a if condition else b before for chooses a value for every item and keeps the length.
• A dict comprehension keeps one value per key, so a key that comes again replaces the earlier value.
• Several for clauses nest from left to right and give one flat list, while a comprehension inside the expression gives a list of lists.
• A comprehension’s loop variables stay inside it, names bound with := are the exception, and in a class body only the first iterable can see the class’s names.
• Round brackets make a generator expression, which produces items on demand in a few hundred bytes and can be read only once, and a for loop is the better tool once the work needs statements or side effects.