Python Module 10 – Iterators, generators and functional tools
Generators and generator expressions
Write generator functions with yield, use generator expressions instead of lists, and measure the memory a generator saves on a million items.
What you will learn
- Write generator functions with yield and predict when their body runs
- Use generator expressions to feed sum(), any() and loops without building lists
- Measure the memory a list and a generator need for the same work
- Recover a generator's return value and recognise the bugs generators cause
- Choose between a list and a generator from how the values are used
Before you start
What you’ll learn · Before you start
What you will learn
- Write generator functions with yield and predict when their body runs
- Use generator expressions to feed sum(), any() and loops without building lists
- Measure the memory a list and a generator need for the same work
- Recover a generator's return value and recognise the bugs generators cause
- Choose between a list and a generator from how the values are used
Before you start
On this page
Adding up a million squares through a list holds every square at once, about 40 MB at the peak; through a generator
expression it holds one at a time, and the peak rounds to 0 KiB. A
generator function: A function whose body contains yield. Calling it runs none of the body and returns a generator; the body then runs a step at a time, up to the next yield, each time the generator is asked for an item. is a function whose body contains yield. Calling it runs
none of the body: it returns a generator: The iterator a generator function returns. Each next() resumes the body where it paused, with its local variables kept, and when the body ends the generator raises StopIteration. Like every iterator, it is used up after one pass., an iterator that runs the body only when
something asks for the next value.
A generator expression: A comprehension in round brackets, such as (x * x for x in xs). It builds no list: it makes each item only when asked for the next one, and it can be read through only once. More in “List, dict and set comprehensions” gives the same laziness in one line:
sum(x * x for x in xs) adds the squares one at a time without building a list of them. Generators matter whenever
data is large, slow to produce or endless, such as the lines of a big log file.
A generator function runs its body in steps, pausing at every yield
Each next() runs the body up to the following yield, hands that value out and pauses, keeping every local
variable, until the next request resumes it from the same spot. The prints inside the body show when it runs:
def countdown(n):
print(" body starts")
while n > 0:
print(f" yield {n}")
yield n
n -= 1
print(" body ends")
gen = countdown(2)
print("created a", type(gen).__name__, "- nothing has run yet")
print("got", next(gen))
print("got", next(gen))
try:
next(gen)
except StopIteration:
print("StopIteration")Output
created a generator - nothing has run yet body starts yield 2 got 2 yield 1 got 1 body ends StopIteration
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 pause_resume.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
A generator and its caller take turns
Text description of the diagram
The diagram has two columns, the caller on the left and the body of countdown(2) on the right, with time running downwards.
- The caller calls next(). The body starts, runs until yield 2, hands 2 back to the caller and pauses with its local variables kept.
- The caller calls next() again. The body continues from the pause, runs until yield 1, hands 1 back and pauses again.
- The caller calls next() a third time. The body continues and reaches its end, which raises StopIteration in the caller.
Creating the generator printed nothing from the body. The first next() ran it from the top to yield 2, and the
second next() resumed it right after that yield, with n still holding 2, until yield 1. The third request let
the loop finish, the body reached its end, and Python raised StopIteration. Step through the run and watch n:
The drawing shows the whole scene; its 7 steps are listed below.
Every step as text
- Created, not started. gen = countdown(2) creates a generator and runs none of the body, so nothing is printed from it yet.
- The first next(). next(gen) starts the body. It prints "body starts" and "yield 2", and reaches yield n with n = 2.
- 2 comes out. yield hands 2 to the caller and pauses the body on that line, with n still 2. The caller prints "got 2".
- The second next(). next(gen) resumes the body right after the yield. It runs n -= 1, goes round the while loop and prints "yield 1".
- 1 comes out. yield hands 1 to the caller and pauses the body again. The caller prints "got 1".
- The third next(). The body resumes, n becomes 0 and the while loop ends. The body prints "body ends" and reaches the end of the function.
- StopIteration. A body that ends raises StopIteration in the caller. This program catches it and prints "StopIteration"; a for loop would simply stop.
Generators keep the iteration protocol of the previous lesson: a generator is its own iterator and is used up after
one pass, like map() or a file.
Checkpoint
Check: Pausing at every yield
A generator expression is a generator written in one line
A generator function suits logic with several steps. A generator expression suits one filter or transformation, and both can be combined, as in this log of a payment service:
import io
LOG = io.StringIO(
"10:00:01 INFO payment 101 started\n"
"10:00:02 ERROR payment 101 failed code=TIMEOUT\n"
"10:00:02 INFO payment 102 started\n"
"10:00:03 INFO payment 102 done\n"
"10:00:05 ERROR payment 103 failed code=DECLINED\n"
"10:00:07 ERROR payment 104 failed code=TIMEOUT\n"
)
def error_lines(lines):
"""Yield only the lines that report an error, one at a time."""
for line in lines:
if " ERROR " in line:
yield line.rstrip("\n")
for line in error_lines(LOG):
print(line)
LOG.seek(0) # back to the start of the "file"
codes = (line.rsplit("code=", 1)[1] for line in error_lines(LOG))
print(sorted(set(codes)))Output
10:00:02 ERROR payment 101 failed code=TIMEOUT 10:00:05 ERROR payment 103 failed code=DECLINED 10:00:07 ERROR payment 104 failed code=TIMEOUT ['DECLINED', 'TIMEOUT']
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 error_lines.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
error_lines() reads the file object line by line and yields only the lines that report an error, so a log of any
size passes through it, one line at a time. The generator expression on the last lines builds on it:
it takes each error line from the generator and keeps the code at its end. Nothing ran until set() asked codes for
its first item. Step through those two lines, one log line at a time:
The drawing shows the whole scene; its 8 steps are listed below.
Every step as text
- Built, not run. The line codes = (… for line in error_lines(LOG)) calls error_lines(LOG), which only creates a generator, and creates the generator expression around it. No log line has been read.
- set() asks for an item. set(codes) asks codes for its first item, codes asks error_lines(), and error_lines() reads line 1. It is an INFO line, so its loop goes on to the next line.
- The first error line. Line 2 reports an error, so error_lines() yields it and pauses. The generator expression takes the text after code=.
- TIMEOUT reaches the set. codes hands TIMEOUT to set(), which keeps it and asks codes for the next item.
- Two INFO lines. error_lines() resumes, reads lines 3 and 4 and passes neither on, because both are INFO lines.
- DECLINED. Line 5 is an error line, and its code, DECLINED, travels through codes into the set.
- TIMEOUT again. Line 6 gives TIMEOUT once more. A set keeps one of each value, so it still holds two codes.
- The end. The log has no line 7, so error_lines() ends, codes ends, and set() is complete. sorted() then puts the two codes in order.
Generator expressions are written in round brackets, and the brackets can be dropped when the expression is the only argument of a call.
Checkpoint
Check: Generator expressions
A generator holds one value at a time: a million squares, measured
The standard library’s tracemalloc traces the memory blocks Python allocates and records the peak. Adding a million
squares through a list and through a generator expression gives the same total:
import tracemalloc
def peak_kib(compute):
"""Run compute() and return its result and the peak memory it allocated, in KiB."""
tracemalloc.start()
result = compute()
peak = tracemalloc.get_traced_memory()[1]
tracemalloc.stop()
return result, peak / 1024
n = 1_000_000
a, list_peak = peak_kib(lambda: sum([x * x for x in range(n)]))
b, gen_peak = peak_kib(lambda: sum(x * x for x in range(n)))
print("same total:", a == b)
print(f"list comprehension: {list_peak:9,.0f} KiB at the peak")
print(f"generator expression: {gen_peak:9,.0f} KiB at the peak")Output
same total: True list comprehension: 39,500 KiB at the peak generator expression: 0 KiB at the peak
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 memory_peak.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Version note
The recorded numbers come from 64-bit CPython, where each reference in a list takes 8 bytes and each of these integers 32 bytes. The browser runs a 32-bit build of Python with smaller objects, so it reports a smaller peak for the list; the generator stays close to zero on both.
In your browser, Pyodide 314.0.7 (CPython 3.14.2) prints:
same total: True list comprehension: 23,527 KiB at the peak generator expression: 0 KiB at the peak
The list comprehension held all million integers at once, about 39,500 KiB or 40 MB at the peak, before sum() read
the first one. The generator expression produced each square, handed it to sum() and let it go, so its peak rounds
to 0 KiB: it needs memory for one value at a time, however many there are. Generator expressions and generator
functions share that property, because both produce one value per request. Numbers that must stay in memory fit in an
array from the array module, with no object per value.
Checkpoint
Check: One value at a time in memory
An endless generator is safe while its consumer stops asking
A generator does not have to end. One that counts for ever is safe while its consumer stops asking:
from itertools import count, islice
def order_ids(prefix="ORD"):
"""An endless supply of order numbers."""
for n in count(1):
yield f"{prefix}-{n:05d}"
ids = order_ids()
print(list(islice(ids, 3))) # the first three
print(next(ids)) # carries on where islice stopped
print(list(islice(order_ids("RET"), 2)))Output
['ORD-00001', 'ORD-00002', 'ORD-00003'] ORD-00004 ['RET-00001', 'RET-00002']
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 order_ids.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
itertools.islice(ids, 3) takes three items and stops, and the next request carries on with number 4, because the
paused generator kept its counter. list(order_ids()) without islice would never return. Step through the
requests and watch the counter wait between them:
The drawing shows the whole scene; its 8 steps are listed below.
Every step as text
- Created. ids = order_ids() creates the generator. Its loop over count(1) has not started, so no number has been made.
- islice() asks. list(islice(ids, 3)) asks islice() for items, and islice() asks the generator for its first one.
- The first number. The generator takes 1 from count(1), yields ORD-00001 and pauses with n = 1.
- Two more. Two more requests give ORD-00002 and ORD-00003, and the generator pauses with n = 3.
- islice() stops asking. islice() has given its three items, so it asks for no more. The generator stays paused, and list() returns the three numbers.
- next(ids). next(ids) asks the same generator again. It resumes where it paused, so count(1) gives 4.
- ORD-00004. The generator yields ORD-00004 and pauses with n = 4. It has made four numbers, one for each request.
- A new generator. order_ids("RET") is a new generator with a count of its own, so its two numbers start again at 1, RET-00001 and RET-00002.
The last line calls order_ids("RET"), which makes a second generator with a counter of its own, so its numbers start
again at 1 while ids stays paused at 4.
Collecting an endless generator
list(order_ids()), sorted(order_ids()) and max(order_ids()) each try to read every item, so none of them
ever returns. Take a fixed number with islice(), or leave a loop with break.
Checkpoint
Check: Endless generators
Three ways to write an iterator, and when each fits
The Countdown class of the previous lesson needed three methods; as a generator function it takes four lines, and as
a generator expression one:
class Countdown:
"""The iterator class of the previous lesson."""
def __init__(self, start):
self.current = start
def __iter__(self):
return self
def __next__(self):
if self.current <= 0:
raise StopIteration
self.current -= 1
return self.current + 1
def countdown(start):
"""The same countdown as a generator function."""
while start > 0:
yield start
start -= 1
print(list(Countdown(3)))
print(list(countdown(3)))
print(list(n for n in range(3, 0, -1)))Output
[3, 2, 1] [3, 2, 1] [3, 2, 1]
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 three_ways.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
All three give [3, 2, 1]. The generator function keeps its place in a paused local variable, and the end of its body
ends it: Python raises the StopIteration that the class had to raise by hand.
| Criterion | Iterator class | Generator function | Generator expression |
|---|---|---|---|
| You write | __iter__() and __next__() | a def with yield | one expression in brackets |
| State is kept in | attributes you update | local variables, paused | the loop it describes |
| Extra methods or attributes | any you need | none of your own | none |
| Good for | iterators with an interface | multi-step logic, endless series | one filter or transformation |
| When to choose | You need methods beyond next() | Most custom iteration | Feeding sum(), any(), max() or a loop |
Generators have a few more abilities. A return statement ends a generator, and its value travels inside the
StopIteration exception. inspect.getgeneratorstate() tells whether a generator is created, running, suspended or
closed. A StopIteration raised inside a generator’s body becomes a RuntimeError, since PEP 479, so it cannot end a
loop silently.
Checkpoint
Check: Choosing the form of an iterator
Check yourself
Practice that adapts to you
Practise this lesson
11 questions about 12 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
How a generator runs, with four bugs reproduced Pro: opens with a Pro, Premium or Ultimate pass
A generator is an object with a state, a paused frame and a way to hand back one final value. This part traces its states, recovers a return value in two ways and reproduces the four bugs that generators cause most often.
In this part:
- Four states, traced from created to closed
- Return values travel inside StopIteration
- Four bugs reproduced: late checks, a stray StopIteration, late binding and an always-true object
- Moving the checks to the call: the fix, run
2 diagrams · 5 runnable examples (Python) · about 12 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Memory and work, measured: when a generator wins and when a list is better Pro: opens with a Pro, Premium or Ultimate pass
A generator saves memory only when the values are used once, and it saves work only when the consumer can stop early. This part measures the memory at four sizes and counts the work, sets out when a list is the better choice and ends with an exercise.
In this part:
- Peak memory at four sizes, list against generator
- Work counted: early exit, and the cost of reading twice
- Sizing a list before you build it, with numbers
- Choosing between a list and a generator, with the trade-offs in a table
- Exercise: chunks of any iterable, and primes that never end
- Chunks of any iterable, and primes that never end
3 runnable examples (Python) · 1 exercise with sample tests · about 13 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Interview questions
-
Fresher to mid · Warm-up
What does calling a generator function return, and when does its body run?
Calling a generator function, a function whose body contains yield, runs none of its body. It returns a generator object, which is an iterator. The body starts at the first next() call, or the first step of a for loop, and runs until it reaches a yield: the yielded value is handed to the caller and the function pauses with its local variables kept. Each later next() resumes it right after that yield. When the body finishes, the generator raises StopIteration, which ends a for loop quietly. Two consequences matter in practice: code at the top of the body, such as argument checks or opening a file, runs only when the first value is requested, and a generator, like every iterator, is used up after one pass.
-
Mid to senior
read_batches(path, size) is a generator function that starts by raising ValueError when size is below 1. A job calls it with size 0 at start-up, but the error appears twenty minutes later in a worker thread, and the traceback never shows the bad call. Explain what happened, and fix the function and its tests.
-
Mid to senior
A nightly report reads 50 million amounts from a database and prints their total, their average and the 10 largest. To save memory, a colleague replaced every list with a generator. Now the average and the top 10 are wrong, with no error. Explain why, estimate the memory involved, and design a version that is correct and memory-light.
-
Mid to senior
A generator that reads records with line = next(it) inside a while True loop worked for years. After a Python upgrade, every run now ends with RuntimeError: generator raised StopIteration. What changed, why was the change made, and how do you fix the code?
Key takeaways
- A function with
yieldreturns a generator, whose body runs only when values are requested and pauses at eachyieldwith its local variables kept. - A generator expression,
(expr for x in xs), is the one-line form; drop its brackets when it is a call’s only argument. - A generator holds one value at a time: a million squares peaked at about 40 MB through a list and under 1 KiB through a generator.
- Generators are iterators, used up after one pass and cut with
itertools.islicewhen they never end. - Choose a generator function for multi-step or endless iteration, a generator expression for one transformation, and an iterator class when the object needs methods of its own.
Terms in this lesson
- generator
- The iterator a generator function returns. Each next() resumes the body where it paused, with its local variables kept, and when the body ends the generator raises StopIteration. Like every iterator, it is used up after one pass.
- generator function
- A function whose body contains yield. Calling it runs none of the body and returns a generator; the body then runs a step at a time, up to the next yield, each time the generator is asked for an item.
References
- The Python Tutorial: Generators (Python Software Foundation)
- The Python Tutorial: Generator Expressions (Python Software Foundation)
- The Python Language Reference: Yield expressions (Python Software Foundation)
- The Python Language Reference: Generator expressions (Python Software Foundation)
- Functional Programming HOWTO: Generators (Python Software Foundation)
- inspect.getgeneratorstate() (Python Software Foundation)
- tracemalloc: Trace memory allocations (Python Software Foundation)
- sys.getsizeof() (Python Software Foundation)
- array: Efficient arrays of numeric values (Python Software Foundation)
- PEP 255: Simple Generators (Python Software Foundation)
- PEP 289: Generator Expressions (Python Software Foundation)
- PEP 479: Change StopIteration handling inside generators (Python Software Foundation)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress