Elixir & Erlang/OTP Module 5 – Recursion, collections and data modelling
Streams and lazy pipelines
Use Elixir streams to read large files and endless sequences in little memory: lazy versus eager, iterate, unfold, resource, and when Enum is the better choice.
What you will learn
- Explain how a lazy Stream pipeline differs from an eager Enum one, and count the work each does
- Predict when the side effects inside a stream run, and how many times
- Build endless and paginated sequences with Stream.iterate/2, Stream.unfold/2, Stream.cycle/1 and Stream.resource/3
- Read a file larger than you want in memory line by line with File.stream!/1
- Choose between Enum and Stream for a pipeline from its input and what it needs
Before you start
What you’ll learn · Before you start
What you will learn
- Explain how a lazy Stream pipeline differs from an eager Enum one, and count the work each does
- Predict when the side effects inside a stream run, and how many times
- Build endless and paginated sequences with Stream.iterate/2, Stream.unfold/2, Stream.cycle/1 and Stream.resource/3
- Read a file larger than you want in memory line by line with File.stream!/1
- Choose between Enum and Stream for a pipeline from its input and what it needs
Before you start
On this page
Counting the status codes of a 19 MB log by reading the whole file first made the VM’s memory grow by about 255 MB; reading the same file as a stream, one line at a time, grew it by under 2 MB, for the same counts. A stream is a recipe for producing elements one at a time, and nothing in it runs until something asks for the elements.
Below, you will count how much work an Enum pipeline and a Stream pipeline do, see when a stream’s side effects run, build endless and paginated sequences, read a large file line by line, and decide when Enum is still the better choice. Elixir 1.20.4 and Erlang/OTP 29.1.1 produced every result on this page.
Enum finishes each step, a stream takes one element through all of them
The Stream documentation draws the line in one sentence: “We say the functions in Stream are lazy and the functions
in Enum are eager.” lazy.exs runs the same pipeline both ways, with a counter from :counters inside the function
to count its calls, then prints each step as it runs:
# The same pipeline with Enum and with Stream: how many times does it call the
# function, and in which order do the steps run?
defmodule Counted do
# Wraps fun so that every call adds 1 to a counter; returns it and a reader.
def wrap(fun) do
counter = :counters.new(1, [])
{fn x -> :counters.add(counter, 1, 1); fun.(x) end, fn -> :counters.get(counter, 1) end}
end
end
{triple, calls} = Counted.wrap(&(&1 * 3))
first = 1..1_000_000 |> Enum.map(triple) |> Enum.take(3)
IO.puts("Enum.map then take 3: #{inspect(first)} after #{calls.()} calls")
{triple, calls} = Counted.wrap(&(&1 * 3))
first = 1..1_000_000 |> Stream.map(triple) |> Enum.take(3)
IO.puts("Stream.map then take 3: #{inspect(first)} after #{calls.()} calls")
IO.puts("Enum, step by step:")
[1, 2, 3] |> Enum.map(&IO.inspect(&1, label: " first step")) |> Enum.map(&IO.inspect(&1 * 3, label: " second step"))
IO.puts("Stream, element by element:")
[1, 2, 3] |> Stream.map(&IO.inspect(&1, label: " first step")) |> Stream.map(&IO.inspect(&1 * 3, label: " second step")) |> Enum.to_list()from session import run
run("""
elixir lazy.exs
""")Output
$ elixir lazy.exs Enum.map then take 3: [3, 6, 9] after 1000000 calls Stream.map then take 3: [3, 6, 9] after 3 calls Enum, step by step: first step: 1 first step: 2 first step: 3 second step: 3 second step: 6 second step: 9 Stream, element by element: first step: 1 second step: 3 first step: 2 second step: 6 first step: 3 second step: 9
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 lazy_session.py
Both versions returned [3, 6, 9]. Enum.map/2 called the function 1,000,000 times, because it finishes its whole
step before Enum.take/2 can say that three are enough. Stream.map/2 did no work at all until Enum.take/2 asked for
elements, then called the function 3 times. The printed order shows how: Enum ran the first step on 1, 2 and 3, then
the second step on all three, while the stream: A lazy enumerable: a recipe for producing elements one at a time that does its work only when an Enum function or Stream.run/1 consumes it, and again each time it is consumed. Stream.map/2, File.stream!/1 and ranges give streams. took 1 through both steps, then 2, then 3.
Step through the two:
The drawing shows the whole scene; its 5 steps are listed below.
Every step as text
- Enum's first step. Enum.map/2 runs the first step on every element before anything else happens: "first step" printed 1, 2 and 3, and the result is a finished list of three.
- Enum's second step. Only then does the second Enum.map/2 start, on that whole list: "second step" printed 3, 6 and 9, and a second list is built.
- Stream, element 1. The stream takes 1 through both steps before it looks at 2: "first step: 1", then "second step: 3". No list is built between the steps.
- Elements 2 and 3. 2 goes through both steps, giving 6, then 3, giving 9. The printed lines alternate, one element at a time.
- Stopping early. That is why Enum.take(3) on Stream.map over a million numbers called the function 3 times: nothing was asked for the fourth element, so it never ran. Enum.map ran it a million times.
The guide on enumerables and streams describes the eager way as “each operation is going to generate an intermediate list until we reach the result”. A stream builds no list between its steps; it composes them, and the work happens element by element when an Enum function at the end consumes it. This is lazy evaluation: Working an element out only when something asks for it. A lazy pipeline takes each element through all its steps before the next, so it can stop early and never builds the lists between its steps; Enum, which finishes each step first, is eager..
Key idea
A Stream function only adds a step to a recipe; the Enum function at the end runs the recipe one element at a time, so a stream does the work for as many elements as are asked for, and no more.
Checkpoint
Check: Lazy streams and eager Enum
A stream runs each time it is consumed
Because a stream is a recipe, the work inside it, and any side effect, happens when it is consumed, and happens again at every consumption. The Stream documentation says “The computations are only performed when you call a function from the Enum module.” Watch a stream that prints as it works:
from session import shell
# A stream with a side effect: built, consumed twice, then run for its side effect alone.
shell([
'noisy = Stream.map([1, 2], fn x -> IO.puts("working on #{x}"); x * 10 end)',
"is_list(noisy)",
"Enum.to_list(noisy)",
"Enum.sum(noisy)",
"Stream.run(noisy)",
"System.halt()",
])Output
$ iex
Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:18:18] [ds:18:18:10] [async-threads:1] [jit]
Interactive Elixir (1.20.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> noisy = Stream.map([1, 2], fn x -> IO.puts("working on #{x}"); x * 10 end)
#Stream<[enum: [1, 2], funs: [#Function<49.48511117/1 in Stream.map/2>]]>
iex(2)> is_list(noisy)
false
iex(3)> Enum.to_list(noisy)
working on 1
working on 2
[10, 20]
iex(4)> Enum.sum(noisy)
working on 1
working on 2
30
iex(5)> Stream.run(noisy)
working on 1
working on 2
:ok
iex(6)> System.halt()
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 recipe_session.py
Building noisy printed nothing but the stream itself, a struct holding the list and the function, and is_list/1
said it is not a list. Enum.to_list/1 ran the function for both elements, and Enum.sum/1 ran it for both again: a
stream keeps no results between consumers. Stream.run/1 consumed it once more and returned :ok; its documentation
says it “is useful when a stream needs to be run, for side effects, and there is no interest in its return result”.
Follow the four lines:
The drawing shows the whole scene; its 4 steps are listed below.
Every step as text
- A recipe. Stream.map/2 returned a %Stream{} holding the list and the function, and printed nothing: no element has been worked on yet. It is not a list, as is_list/1 said.
- The first consumer. Enum.to_list/1 asks for every element, so the function runs for 1 and for 2, printing two lines, and the list [10, 20] comes back.
- The second consumer. Enum.sum/1 asks for the elements again, and the stream does the work again: two more lines printed. A stream keeps no results between consumers.
- Only the side effect. Stream.run/1 consumes the stream and throws the results away, returning :ok. The documentation names it for streams run for their side effects alone.
A stream with side effects, consumed twice
emails = Stream.map(users, &send_reminder/1) followed by Enum.count(emails) and Enum.to_list(emails) sends
every reminder twice. Consume a stream with side effects exactly once, with Stream.run/1 or one Enum call, or turn
it into a list once with Enum.to_list/1 and use the list afterwards.
Checkpoint
Check: When a stream runs
Endless sequences with iterate, unfold and cycle
A stream can describe a sequence with no end, because only the elements that are asked for are ever made.
Stream.iterate/2 starts from a value and applies a function to get each next one, Stream.unfold/2 builds each
element from a state and decides when to stop, and Stream.cycle/1 repeats a collection forever. The session builds
retry delays that double each time, then shares tickets among two people in turn:
from session import shell
# Sequences without an end, cut by take/2 or by the stream itself.
shell([
"Stream.iterate(100, &(&1 * 2)) |> Enum.take(6)",
"Stream.unfold(100, fn delay when delay > 5_000 -> nil; delay -> {delay, delay * 2} end) |> Enum.to_list()",
'tickets = ["T-101", "T-102", "T-103", "T-104", "T-105"]',
'Enum.zip(tickets, Stream.cycle(["Asha", "Dev"]))',
"Stream.cycle([]) |> Enum.take(2)",
"System.halt()",
])Output
$ iex
Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:18:18] [ds:18:18:10] [async-threads:1] [jit]
Interactive Elixir (1.20.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Stream.iterate(100, &(&1 * 2)) |> Enum.take(6)
[100, 200, 400, 800, 1600, 3200]
iex(2)> Stream.unfold(100, fn delay when delay > 5_000 -> nil; delay -> {delay, delay * 2} end) |> Enum.to_list()
[100, 200, 400, 800, 1600, 3200]
iex(3)> tickets = ["T-101", "T-102", "T-103", "T-104", "T-105"]
["T-101", "T-102", "T-103", "T-104", "T-105"]
iex(4)> Enum.zip(tickets, Stream.cycle(["Asha", "Dev"]))
[
{"T-101", "Asha"},
{"T-102", "Dev"},
{"T-103", "Asha"},
{"T-104", "Dev"},
{"T-105", "Asha"}
]
iex(5)> Stream.cycle([]) |> Enum.take(2)
** (ArgumentError) cannot cycle over an empty enumerable
(elixir 1.20.4) lib/stream.ex:1403: Stream.cycle/1
iex:5: (file)
iex(5)> System.halt()
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 endless_session.py
Stream.iterate(100, &(&1 * 2)) never ends, so Enum.take/2 cut it after six delays. Stream.unfold/2 ended by
itself: its function answers {element, next_state} to go on, and the documentation says “The enumeration finishes if
it returns nil”, which the clause delay when delay > 5_000 did. Stream.cycle/1 gave Asha and Dev in turn, and
Enum.zip/2 stopped when the five tickets ran out. Step through the unfold:
The drawing shows the whole scene; its 5 steps are listed below.
Every step as text
- The first element. Stream.unfold/2 calls the function with the starting state, 100. The answer {100, 200} means: emit 100, and call me next with 200.
- Doubling. Each call emits its state and doubles it for the next call: 200, 400, 800 and 1600 are emitted in turn.
- The last delay. With the state 3200 the function answers {3200, 6400}: 3200 is emitted, the last delay under 5,000.
- nil ends it. With 6400 the first clause matches, delay > 5_000, and the function answers nil. The documentation says the enumeration finishes when it returns nil.
- The result. Enum.to_list/1 collected six delays. Stream.iterate/2 with the same doubling never ends, which is why it was cut with Enum.take/2.
An empty cycle raises
Stream.cycle([]) raised ArgumentError, “cannot cycle over an empty enumerable”: with nothing to repeat, the cycle
refuses rather than become a stream that never gives an element. A round robin over a list that can be empty needs
its own check for [] first.
Checkpoint
Check: Endless sequences: iterate, unfold, cycle
Read a large file one line at a time
File.stream!/1 returns a stream of a file’s lines, and the File documentation says that “the file is automatically
opened each time streaming begins”. log_session.py writes a log of a million lines, 19 MB, then counts its status
codes twice, each in a VM of its own, while a second process samples the VM’s total memory every millisecond:
# Writes access.log: 1,000,000 lines such as "GET /menu 200 412ms", made from a
# fixed seed so every run writes the same file.
defmodule LogGen do
@paths {"/", "/menu", "/cart", "/pay", "/orders", "/help"}
@statuses {200, 200, 200, 200, 304, 404, 500}
def write(path, lines) do
:rand.seed(:exsss, {7, 7, 7})
# :delayed_write gathers small writes into larger ones.
File.open!(path, [:write, :delayed_write], fn file ->
for _ <- 1..lines//1 do
path = elem(@paths, :rand.uniform(6) - 1)
status = elem(@statuses, :rand.uniform(7) - 1)
IO.binwrite(file, ["GET ", path, " ", Integer.to_string(status), " ", Integer.to_string(:rand.uniform(900)), "ms\n"])
end
end)
end
end
LogGen.write("access.log", 1_000_000)
IO.puts("access.log: #{File.stat!("access.log").size} bytes")# Counts the status codes of access.log, either by reading the whole file
# (elixir count_statuses.exs read) or by streaming it line by line
# (elixir count_statuses.exs stream), and reports how much the VM's memory grew.
defmodule Peak do
# Runs fun in a new process while another samples the VM's total memory every
# millisecond; returns fun's result and the largest growth seen, in bytes.
def measure(fun) do
:erlang.garbage_collect()
base = :erlang.memory(:total)
parent = self()
sampler = spawn(fn -> sample(base) end)
result = Task.async(fun) |> Task.await(:infinity)
send(sampler, {:stop, parent})
receive do
{:peak, peak} -> {result, peak - base}
end
end
defp sample(peak) do
receive do
{:stop, from} -> send(from, {:peak, max(peak, :erlang.memory(:total))})
after
1 -> sample(max(peak, :erlang.memory(:total)))
end
end
end
# "GET /menu 200 412ms" -> "200"
status = fn line -> line |> String.split(" ") |> Enum.at(2) end
{counts, grew} =
case System.argv() do
["read"] ->
Peak.measure(fn ->
File.read!("access.log") |> String.split("\n", trim: true) |> Enum.map(status) |> Enum.frequencies()
end)
["stream"] ->
Peak.measure(fn -> File.stream!("access.log") |> Stream.map(status) |> Enum.frequencies() end)
end
IO.inspect(counts)
IO.puts("memory grew by at most #{:erlang.float_to_binary(grew / 1_000_000, decimals: 1)} MB")from session import run
# One log of a million lines, counted twice: read whole, then streamed.
run("""
elixir gen_log.exs
elixir count_statuses.exs read
elixir count_statuses.exs stream
""")Output
$ elixir gen_log.exs
access.log: 19382219 bytes
$ elixir count_statuses.exs read
%{"200" => 571228, "304" => 142969, "404" => 142588, "500" => 143215}
memory grew by at most 255.4 MB
$ elixir count_statuses.exs stream
%{"200" => 571228, "304" => 142969, "404" => 142588, "500" => 143215}
memory grew by at most 1.1 MB
This output changes from run to run: the memory the sampler sees changes a little from run to run, with what the VM allocates while it samples
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 log_session.py
Both counts were the same, 571,228 lines with 200 and so on. Reading the whole file first made memory grow by about 255 MB, more than 13 times the size of the file: the whole text, a million line strings and the list holding them, then the list of statuses, all alive at once. The stream held one line at a time, and memory grew by under 2 MB. Follow one line through the streamed version:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- Nothing read yet. File.stream!/1 returns a stream without reading anything; the File documentation says the file is opened each time streaming begins.
- One line. Enum.frequencies/1 asks for an element, and the stream reads the next line from the file, here "GET /menu 200 412ms\n".
- The status. Stream.map/2 turns the line into its status code, "200". The line itself is now garbage, and only the code goes on.
- Counted. Enum.frequencies/1 adds 1 to the count of "200". The map never holds more than the four status codes, however long the file.
- A million times. The same three stages run for each of the 1,000,000 lines, so memory grew by under 2 MB, where reading the whole file first grew it by about 255 MB.
- Closed. At the end of the file the stream closes it and frequencies/1 returns the four counts, the same as the version that read the whole file.
File.stream!/2 can also cut a file into pieces of a fixed number of bytes instead of lines, which suits binary data.
The second argument says which, and the modes come third:
# A short log of two lines, read as lines, then in pieces of 8 bytes, then with
# the argument order that Elixir 1.20 deprecates.
File.write!("short.log", "GET /menu 200 412ms\nGET /pay 500 37ms\n")
IO.inspect(File.stream!("short.log") |> Enum.to_list(), label: "lines")
IO.inspect(File.stream!("short.log", 8) |> Enum.take(3), label: "8 bytes")
IO.inspect(File.stream!("short.log", [], :line) |> Enum.count(), label: "old order")from session import run
run("""
elixir chunks.exs
""")Output
$ elixir chunks.exs lines: ["GET /menu 200 412ms\n", "GET /pay 500 37ms\n"] 8 bytes: ["GET /men", "u 200 41", "2ms\nGET "] warning: File.stream!(path, modes, line_or_byte) is deprecated, invoke File.stream!(path, line_or_bytes, modes) instead (elixir 1.20.4) lib/file.ex:2256: File.stream!/3 chunks.exs:6: (file) (elixir 1.20.4) src/elixir_compiler.erl:102: :elixir_compiler.dispatch/4 (elixir 1.20.4) src/elixir_compiler.erl:76: :elixir_compiler.compile/4 (elixir 1.20.4) src/elixir_compiler.erl:42: :elixir_compiler.optimize_defmodule/2 (elixir 1.20.4) src/elixir_lexical.erl:18: :elixir_lexical.run/3 old order: 2
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 chunks_session.py
Elixir 1.20: modes go last
The 1.20 changelog deprecates File.stream!(path, modes, lines_or_bytes) in favour of
File.stream!(path, lines_or_bytes, modes). The old order still worked, but printed the warning above with the
place it was called from.
Checkpoint
Check: Files as streams
A resource stream wraps anything you open and close
Files, sockets and paginated APIs share a shape: open, read in pieces, close. Stream.resource/3 turns that shape into
a stream, with three functions: one that opens the source and returns an accumulator, one that answers
{elements, accumulator} or {:halt, accumulator}, and one that closes it, which the documentation says runs “at the
end of enumeration (both in cases of success and failure)”. Shop.orders/0 wraps a source that returns three orders a
page:
# A paginated source, such as an API that returns 3 orders a page, wrapped in
# Stream.resource/3 so that callers see one stream of orders.
defmodule Shop do
@orders Enum.map(1..8, &"order-#{&1}")
# The orders of one page; [] after the last page.
def fetch(page) do
IO.puts(" fetching page #{page}")
Enum.slice(@orders, (page - 1) * 3, 3)
end
def orders do
Stream.resource(
# start: open the source; here, remember which page comes next.
fn -> 1 end,
# next: one page of elements, or :halt when there are no more.
fn page ->
case fetch(page) do
[] -> {:halt, page}
orders -> {orders, page + 1}
end
end,
# after: close the source, whether the stream ended or was stopped.
fn page -> IO.puts(" closed, next page would have been #{page}") end
)
end
end
IO.puts("Take 4 orders:")
IO.inspect(Shop.orders() |> Enum.take(4))
IO.puts("Count every order:")
IO.inspect(Shop.orders() |> Enum.count())from session import run
run("""
elixir pages.exs
""")Output
$ elixir pages.exs Take 4 orders: fetching page 1 fetching page 2 closed, next page would have been 3 ["order-1", "order-2", "order-3", "order-4"] Count every order: fetching page 1 fetching page 2 fetching page 3 fetching page 4 closed, next page would have been 4 8
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 pages_session.py
Taking four orders fetched two pages, because the fourth order is on page 2, and closed the source at once; page 3 was never fetched. Counting every order fetched pages until page 4 came back empty, then closed. Step through the first run:
The drawing shows the whole scene; its 5 steps are listed below.
Every step as text
- Start. Nothing happens until Enum.take/2 asks for the first order. Then the start function runs once and returns the accumulator, page 1.
- Page 1. The next function fetches page 1 and answers {orders, 2}: three orders to emit, and page 2 for the next call. take/2 has 3 of the 4 it wants.
- Page 2. take/2 wants one more, so next runs again and fetches page 2. Its first order, order-4, is the fourth.
- Enough. take/2 has its 4 orders and stops asking. order-5 and order-6 were fetched but not used, and page 3 is never fetched.
- After. The after function runs with the accumulator, page 3, and closes the source: "closed, next page would have been 3". The documentation says it runs after success and failure alike.
This is how File.stream!/1 itself works: the guide notes that it “builds on top of Stream.resource/3 to stream
files”. The same three functions fit a database cursor or a socket, and the after function is where the connection is
released.
Checkpoint
Check: Stream.resource/3
Enum first, and a stream when laziness pays
Laziness has a cost. Each element of a stream passes through the machinery that composes its steps, and work.exs
counts the cost in reductions, the units of work the VM counts for each process, for the same pipeline both ways:
# The same pipeline with Enum and with Stream, in a new process each time:
# the answer, and the work done, counted in reductions (the VM's unit of work).
defmodule Work do
def reductions(fun) do
task =
Task.async(fn ->
{:reductions, before} = Process.info(self(), :reductions)
result = fun.()
{:reductions, later} = Process.info(self(), :reductions)
{result, later - before}
end)
Task.await(task)
end
end
for n <- [1_000, 1_000_000] do
{sum, eager} = Work.reductions(fn -> 1..n |> Enum.map(&(&1 * 3)) |> Enum.filter(&(rem(&1, 2) == 0)) |> Enum.sum() end)
{^sum, lazy} = Work.reductions(fn -> 1..n |> Stream.map(&(&1 * 3)) |> Stream.filter(&(rem(&1, 2) == 0)) |> Enum.sum() end)
IO.puts("n = #{n}: sum #{sum}; Enum #{eager} reductions, Stream #{lazy}, Stream/Enum #{:erlang.float_to_binary(lazy / eager, decimals: 2)}")
endfrom session import run
run("""
elixir work.exs
""")Output
$ elixir work.exs n = 1000: sum 751500; Enum 7151 reductions, Stream 10307, Stream/Enum 1.44 n = 1000000: sum 750001500000; Enum 7188322 reductions, Stream 9527847, Stream/Enum 1.33
This output changes from run to run: the reductions of the larger run vary slightly from run to run, with the garbage collections the process makes
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 work_session.py
The stream did about 1.4 times the work of the Enum pipeline for 1,000 numbers, and about 1.3 times for a million, for the same sums. In exchange it built no lists between its steps. So the Elixir guide’s advice is to “focus on the Enum module first and only move to Stream for the particular scenarios where laziness is required”. Answer two questions about your own data to see which fits:
Answer the questions above: the table then marks what fits your case and says why the rest does not.
| Pipeline | Work per element | Memory on large input | Stops early | Endless or external data |
|---|---|---|---|---|
| Enum all the way Looks like: list |> Enum.map(…) |> Enum.filter(…) |> Enum.sum() | Good 7,151 reductions for 1,000 numbers in work.exs | Poor builds every list: memory grew 255 MB for a 19 MB log | Mixed map then take 3 ran the function 1,000,000 times | Poor cannot finish a step on an endless sequence |
| Stream, then Enum at the end Looks like: list |> Stream.map(…) |> Stream.filter(…) |> Enum.sum() | Mixed about 1.4 times the reductions for the same answer | Good one element at a time: under 2 MB for the same log | Good take 3 ran the function 3 times | Good iterate, cycle, unfold and resource produce on demand |
How the answers decide
- “How big is the input?” Large, endless, or read from outside: rules out Enum all the way. Each Enum step builds its whole result, so a large input is held in memory several times over; a stream holds one element.
- “How big is the input?” Small: it fits in memory easily and “Does the result need every element?” Yes, all of them: adds a caveat to Stream, then Enum at the end. For small data the stream only adds work, about 1.4 times the reductions in work.exs; the Elixir guide suggests Enum first.
- “Does the result need every element?” No, it can stop after a few: adds a caveat to Enum all the way. With Enum every earlier step runs on the whole input before the step that stops; a stream works only as far as needed.
In short: Enum for collections that fit in memory and are used whole, and a stream for large or endless input, for data read from files, sockets and APIs, and for pipelines that stop after the first few results.
Checkpoint
Check: Choosing Enum or Stream
Practise: the most requested paths
LogStats.top_paths/2 takes the path of an access log, whose lines look like those of gen_log.exs, and returns the
n most requested paths with their counts. The log may be larger than memory, so stream it. Copy the two files into an
empty folder:
# Statistics of an access log whose lines look like "GET /menu 200 12ms": the
# method, the path, the status code and the time taken. The log can be larger
# than memory, so read it as a stream.
defmodule LogStats do
# The n most requested paths as {path, count}, the most requested first; paths
# with equal counts in alphabetical order.
def top_paths(_log_path, _n), do: []
endCode.require_file("log_stats.ex", __DIR__)
ExUnit.start(seed: 0)
defmodule LogStatsTest do
use ExUnit.Case
# Each test gets an empty folder of its own, under tmp/ beside this file.
@moduletag :tmp_dir
defp write_log(dir, lines) do
path = Path.join(dir, "access.log")
File.write!(path, Enum.map(lines, &[&1, "\n"]))
path
end
test "the most requested paths, most first", %{tmp_dir: dir} do
log =
write_log(dir, [
"GET /menu 200 12ms",
"GET /cart 200 30ms",
"GET /menu 304 2ms",
"POST /pay 500 81ms",
"GET /menu 200 9ms",
"GET /cart 404 5ms"
])
assert LogStats.top_paths(log, 2) == [{"/menu", 3}, {"/cart", 2}]
end
test "equal counts in alphabetical order of the path", %{tmp_dir: dir} do
log = write_log(dir, ["GET /pay 200 1ms", "GET /cart 200 1ms", "GET /help 200 1ms"])
assert LogStats.top_paths(log, 3) == [{"/cart", 1}, {"/help", 1}, {"/pay", 1}]
end
test "an empty log", %{tmp_dir: dir} do
assert LogStats.top_paths(write_log(dir, []), 5) == []
end
test "a log of 200,000 lines", %{tmp_dir: dir} do
log = write_log(dir, for(i <- 1..200_000, do: "GET /item/#{rem(i, 7)} 200 1ms"))
assert LogStats.top_paths(log, 3) == [{"/item/1", 28572}, {"/item/2", 28572}, {"/item/3", 28572}]
end
endRun elixir log_stats_test.exs. Each test writes its own log into a folder that ExUnit makes for it, under tmp/.
The starter returns [], so only the test of the empty log passes:
from session import run
# The tests against the starter, which returns [] for every log.
run("""
cd practice
!elixir log_stats_test.exs
""")Output
$ cd practice
$ elixir log_stats_test.exs
Running ExUnit with seed: 0, max_cases: 36
1) test the most requested paths, most first (LogStatsTest)
log_stats_test.exs:15
Assertion with == failed
code: assert LogStats.top_paths(log, 2) == [{"/menu", 3}, {"/cart", 2}]
left: []
right: [{"/menu", 3}, {"/cart", 2}]
stacktrace:
log_stats_test.exs:26: (test)
2) test equal counts in alphabetical order of the path (LogStatsTest)
log_stats_test.exs:29
Assertion with == failed
code: assert LogStats.top_paths(log, 3) == [{"/cart", 1}, {"/help", 1}, {"/pay", 1}]
left: []
right: [{"/cart", 1}, {"/help", 1}, {"/pay", 1}]
stacktrace:
log_stats_test.exs:31: (test)
.
3) test a log of 200,000 lines (LogStatsTest)
log_stats_test.exs:38
Assertion with == failed
code: assert LogStats.top_paths(log, 3) == [{"/item/1", 28572}, {"/item/2", 28572}, {"/item/3", 28572}]
left: []
right: [{"/item/1", 28572}, {"/item/2", 28572}, {"/item/3", 28572}]
stacktrace:
log_stats_test.exs:40: (test)
Finished in 0.2 seconds (0.1s on load, 0.00s async, 0.1s sync)
Result: 1/4 passed
Failed: 3 tests
[exit status 2]
This output changes from run to run: ExUnit prints how long the tests took, and max_cases follows the computer's processors
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 practice_session.py
Stream the file with File.stream!/1, map each line to its path, the second word, and let Enum.frequencies/1
consume the stream: the map it builds holds one count per path, however long the log. Then sort the counts with the
tuple key {-count, path}, from the Enum lesson, and take n. A finished version passes all four tests:
from session import run
# The same tests against a finished LogStats module.
run("""
cd finished
cp ../practice/log_stats_test.exs .
elixir log_stats_test.exs
""")Output
$ cd finished $ cp ../practice/log_stats_test.exs . $ elixir log_stats_test.exs Running ExUnit with seed: 0, max_cases: 36 .... Finished in 0.2 seconds (0.04s on load, 0.00s async, 0.2s sync) Result: 4 passed
This output changes from run to run: ExUnit prints how long the tests took, and max_cases follows the computer's processors
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 finished_session.py
Then test what you have learned: the self-check takes the ideas of this lesson one question at a time, starting with the one you know least, and explains each answer once you have chosen:
Practice that adapts to you
Practise this lesson
18 questions about 6 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
Warm-up: what is the difference between Enum and Stream in Elixir?
Enum is eager: each function goes through its whole input and returns a finished result, so a pipeline of Enum calls
builds a full intermediate list at every step. Stream is lazy: its functions return a recipe, a stream, and do no work
until an Enum function or Stream.run/1 consumes it; then each element goes through every step before the next one
starts. So Stream.map/2 followed by Enum.take(3) calls the function 3 times, where Enum.map/2 would call it for
every element, and a file can be processed line by line in little memory. The price is some work per element, about
1.4 times the reductions of the Enum version in this lesson’s measurement, and a stream runs again every time it is
consumed, side effects included. Use Enum by default and streams for large, endless or external data.
Two questions from jobs that read files and send messages:
Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.
-
Mid to senior
A nightly job counts the requests per path in a 4 GB access log with File.read!/1, String.split/2, Enum.map/2 and Enum.frequencies/1, on a node with 8 GB of memory. It started failing as the log grew, the node running out of memory. Rewrite it, and say how you would show that its memory stays bounded.
-
Fresher to mid
A job builds reminders = Stream.map(due_users, &send_reminder/1), logs the number from Enum.count(reminders), then calls Enum.to_list(reminders) to fill an audit table. Customers get every reminder twice. Explain the cause, fix it, and make sure it cannot come back.
Key takeaways
- Enum functions are eager and finish each step for the whole input; Stream functions are lazy and take each element
through every step, so taking 3 from
Stream.map/2over a million numbers called the function 3 times. - A stream is a recipe that runs only when it is consumed, and again every time it is consumed, side effects included;
Stream.run/1consumes one for its side effects alone. Stream.iterate/2andStream.cycle/1never end and are cut with functions such asEnum.take/2, whileStream.unfold/2ends when its function returnsnil; a cycle of an empty list raisesArgumentError.File.stream!/1reads a file line by line, so counting a 19 MB log grew memory by under 2 MB instead of about 255 MB; give the lines-or-bytes argument second and the modes last.Stream.resource/3wraps anything opened, read in pieces and closed, and its after function runs even when the consumer stops early.- A stream costs more work per element, about 1.4 times the reductions here, so use Enum for data that fits in memory and streams for large, endless or external data.
Terms in this lesson
- lazy evaluation
- Working an element out only when something asks for it. A lazy pipeline takes each element through all its steps before the next, so it can stop early and never builds the lists between its steps; Enum, which finishes each step first, is eager.
- stream
- A lazy enumerable: a recipe for producing elements one at a time that does its work only when an Enum function or Stream.run/1 consumes it, and again each time it is consumed. Stream.map/2, File.stream!/1 and ranges give streams.
References
- Stream (Elixir) (The Elixir Team (hexdocs.pm))
- Elixir guide: Enumerables and Streams (The Elixir Team (hexdocs.pm))
- File (Elixir): stream!/3 (The Elixir Team (hexdocs.pm))
- Changelog for Elixir v1.20 (File.stream! argument order) (The Elixir Team (hexdocs.pm))
- erlang (ERTS): memory/1 and process_info/2 (reductions) (Erlang/OTP (erlang.org))
- counters (ERTS) (Erlang/OTP (erlang.org))
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress