Comprehensions with for

for x <- 1..3, y <- [:a, :b], do: {x, y} returns six tuples, every pairing of three numbers with two atoms, from one line that would take two nested Enum calls to write. A comprehension takes elements from one or more collections, keeps the ones that pass its filters and match its patterns, and collects a result for each, into a list or into whatever you ask for.

Below, you will predict the order of the results of several generators, filter elements and skip them by their shape, collect into maps, strings, sets and totals, and take a binary apart three bytes at a time. Elixir 1.20.4 printed every result here, on Erlang/OTP 29.1.1, whose own shell runs the Erlang session at the end of the second section.

Generators give every combination, in nested order

A comprehensioncomprehension: A for expression that takes elements from one or more generators, keeps those that pass its filters, and collects the value of its body for each one: into a list, unless into: or reduce: asks for something else. starts with for and one or more generatorsgenerator: The part of a comprehension written pattern <- collection, which hands its elements one at a time to the rest of the comprehension. Elements that do not match the pattern are skipped; several generators give every combination., each written pattern <- collection. The comprehensions guide puts the rule for several generators in one line: “Multiple generators can also be used to calculate the Cartesian product of two lists”. Try one, two and three kinds of generator:

from session import shell

# One generator, two generators, and a generator that uses the value of the one before it.
shell([
    "for x <- [1, 2, 3], do: x * 100",
    "for x <- 1..3, y <- [:a, :b], do: {x, y}",
    'for size <- ["S", "M"], colour <- ["red", "blue"], do: size <> "-" <> colour',
    "for x <- [1, 2], y <- [x, x * 10], do: y",
    "for x <- [], y <- [:a, :b], do: {x, y}",
    "System.halt()",
])

$ iex
Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:18:18] [ds:18:18:10] [async-threads:1] [jit]

Interactive Elixir (1.20.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> for x <- [1, 2, 3], do: x * 100
[100, 200, 300]
iex(2)> for x <- 1..3, y <- [:a, :b], do: {x, y}
[{1, :a}, {1, :b}, {2, :a}, {2, :b}, {3, :a}, {3, :b}]
iex(3)> for size <- ["S", "M"], colour <- ["red", "blue"], do: size <> "-" <> colour
["S-red", "S-blue", "M-red", "M-blue"]
iex(4)> for x <- [1, 2], y <- [x, x * 10], do: y
[1, 10, 2, 20]
iex(5)> for x <- [], y <- [:a, :b], do: {x, y}
[]
iex(6)> System.halt()

One generator gave one result per element, [100, 200, 300]. Two generators gave six tuples, and the order is the point: for each x the second generator runs from its start, so the last generator varies fastest, as the inner loop of two nested loops would. Step through the six results:

A later generator may use the variables of an earlier one: y <- [x, x * 10] gave [1, 10, 2, 20]. And an empty generator anywhere means no combination at all, so for x <- [], … returned [] without running the body.

That last point has a catch in Elixir 1.20, whose type checker infers the types of variables in code without type annotations. Tickets.print_all/2 uses ticket as an atom inside a comprehension and as a number after it:

# Prints a line for each item, labelled with the ticket, then returns the next
# ticket number. The ticket is used as an atom inside the comprehension and as a
# number after it.
defmodule Tickets do
  def print_all(ticket, items) do
    for item <- items do
      IO.puts("#{Atom.to_string(ticket)}: #{item}")
    end

    ticket + 1
  end
end

from session import run

# Compiling the module shows the type checker's warning; the call with no items still runs.
run("""
elixirc typed.ex
elixir -e "IO.inspect(Tickets.print_all(7, []))"
""")

$ elixirc typed.ex
    warning: incompatible types given to Kernel.+/2:

        ticket + 1

    given types:

        -dynamic(atom())-, integer()

    but expected one of:

        #1
        integer(), integer()

        #2
        integer(), float()

        #3
        float(), integer()

        #4
        float(), float()

    where "ticket" was given the type:

        # type: dynamic(atom())
        # from: typed.ex:7:23
        Atom.to_string(ticket)

    type warning found at:
    │
 10 │     ticket + 1
    │            ~
    │
    └─ typed.ex:10:12: Tickets.print_all/2
$ elixir -e "IO.inspect(Tickets.print_all(7, []))"
8

Filters keep what passes, and a pattern skips what does not match

A filter is an ordinary expression written after a generator: the for/1 documentation says “Filters must evaluate to truthy values (everything but nil and false)”, and elements for which a filter returns false or nil are dropped. A generator’s pattern filters too, by shape. Try both:

# The amount of a successful payment result; there is no clause for an error.
defmodule Payments do
  def amount({:ok, amount}), do: amount
end

from session import shell

# A filter, patterns in generators, and the same pattern in a function given to Enum.map/2.
shell([
    "for x <- 1..10, rem(x, 3) == 0, do: x * x",
    "IO.inspect(for(x <- 1..10, rem(x, 3) == 0, do: x * x), charlists: :as_lists)",
    "results = [{:ok, 1200}, {:error, :timeout}, {:ok, 450}, {:error, :declined}]",
    "for {:ok, amount} <- results, do: amount",
    "for {:error, reason} <- results, do: reason",
    'for {item, price} when price > 40 <- %{"tea" => 30, "toast" => 45, "idli" => 60}, do: item',
    "Enum.map(results, &Payments.amount/1)",
    "System.halt()",
], command="iex payments.ex")

$ iex payments.ex
Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:18:18] [ds:18:18:10] [async-threads:1] [jit]

Interactive Elixir (1.20.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> for x <- 1..10, rem(x, 3) == 0, do: x * x
~c"\t$Q"
iex(2)> IO.inspect(for(x <- 1..10, rem(x, 3) == 0, do: x * x), charlists: :as_lists)
[9, 36, 81]
~c"\t$Q"
iex(3)> results = [{:ok, 1200}, {:error, :timeout}, {:ok, 450}, {:error, :declined}]
[ok: 1200, error: :timeout, ok: 450, error: :declined]
iex(4)> for {:ok, amount} <- results, do: amount
[1200, 450]
iex(5)> for {:error, reason} <- results, do: reason
[:timeout, :declined]
iex(6)> for {item, price} when price > 40 <- %{"tea" => 30, "toast" => 45, "idli" => 60}, do: item
["idli", "toast"]
iex(7)> Enum.map(results, &Payments.amount/1)
** (FunctionClauseError) no function clause matching in Payments.amount/1

    The following arguments were given to Payments.amount/1:

        # 1
        {:error, :timeout}

    payments.ex:3: Payments.amount/1
    (elixir 1.20.4) lib/enum.ex:1725: Enum."-map/2-lists^map/1-1-"/2
    (elixir 1.20.4) lib/enum.ex:1725: Enum."-map/2-lists^map/1-1-"/2
    iex:7: (file)
iex(7)> System.halt()

The first line returned the squares of 3, 6 and 9, yet IEx printed ~c"\t$Q": [9, 36, 81] is a list of printable codes, which IEx shows as a charlist, as the strings lesson explained, and charlists: :as_lists showed the numbers. results printed in keyword-list form for the same reason of display: it is a list of two-element tuples whose first elements are atoms, which is what a keyword list is.

for {:ok, amount} <- results returned [1200, 450]. The guide states the rule: “all non-matching patterns are ignored”, so the two errors were skipped without a word, and {:error, reason} <- results picked out the other two. A guard after the pattern, when price > 40, filters in the generator itself. The same pattern as the only clause of Payments.amount/1 behaved the other way: given {:error, :timeout} by Enum.map/2, it raised FunctionClauseError. Step through what the generator did:

Erlang has list comprehensions too, with the pattern and the result written the other way round. Its shell on Erlang/OTP 29.1.1 shows two things Elixir’s for does not have:

from session import shell

# Comprehensions in the Erlang shell of Erlang/OTP 29.1.1.
shell([
    "[V || {ok, V} <- [{ok, 1200}, {error, timeout}, {ok, 450}]].",
    "[V || {ok, V} <:- [{ok, 1200}, {error, timeout}, {ok, 450}]].",
    "[X, -X || X <- [1, 2, 3]].",
    "halt().",
], command="erl", kind="erl")

$ erl
Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:18:18] [ds:18:18:10] [async-threads:1] [jit]

Eshell V17.1 (press Ctrl+G to abort, type help(). for help)
1> [V || {ok, V} <- [{ok, 1200}, {error, timeout}, {ok, 450}]].
[1200,450]
2> [V || {ok, V} <:- [{ok, 1200}, {error, timeout}, {ok, 450}]].
** exception error: no match of right hand side value {error,timeout}
3> [X, -X || X <- [1, 2, 3]].
[1,-1,2,-2,3,-3]
4> halt().

With <-, Erlang skips the elements that do not match, as Elixir does. With the strict generator <:- the same comprehension raised on {error, timeout}. And [X, -X || X <- [1, 2, 3]] emitted two elements for each one it took.

into, reduce and uniq decide what a comprehension builds

A comprehension returns a list unless told otherwise. :into takes any collectablecollectable: A value that results can be put into, because its type implements the Collectable protocol, such as a list, a map, a MapSet or a binary. The into: option of for and Enum.into/2 take one.: the for/1 documentation says it “accepts any structure as long as it implements the Collectable protocol”. :reduce turns the body into clauses that receive the accumulator, as Enum.reduce/3 does, and :uniq keeps each result only once:

from session import shell

# What a comprehension builds: a map, a string, a single value, a list without repeats, a set.
shell([
    'menu = %{"tea" => 30, "toast" => 45, "idli" => 60}',
    "for {item, price} <- menu, into: %{}, do: {item, price + 5}",
    'for <<c <- "chai latte">>, c != ?\\s, into: "", do: <<c>>',
    "for price <- [120, 45, 300], reduce: 0, do: (total -> total + price)",
    "for x <- [3, 1, 3, 2, 1], uniq: true, do: x * 100",
    "for x <- 1..6, into: MapSet.new(), do: rem(x, 3)",
    "System.halt()",
])

$ iex
Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:18:18] [ds:18:18:10] [async-threads:1] [jit]

Interactive Elixir (1.20.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> menu = %{"tea" => 30, "toast" => 45, "idli" => 60}
%{"idli" => 60, "tea" => 30, "toast" => 45}
iex(2)> for {item, price} <- menu, into: %{}, do: {item, price + 5}
%{"idli" => 65, "tea" => 35, "toast" => 50}
iex(3)> for <<c <- "chai latte">>, c != ?\s, into: "", do: <<c>>
"chailatte"
iex(4)> for price <- [120, 45, 300], reduce: 0, do: (total -> total + price)
465
iex(5)> for x <- [3, 1, 3, 2, 1], uniq: true, do: x * 100
[300, 100, 200]
iex(6)> for x <- 1..6, into: MapSet.new(), do: rem(x, 3)
MapSet.new([0, 1, 2])
iex(7)> System.halt()

into: %{} built a map from the {item, price} pairs, so the prices rose by 5 and the result stayed a map, the job that needed Map.new/2 in the Enum lesson. A bitstring generator with into: "" removed the space from “chai latte”, one byte at a time. reduce: 0 added up the prices to 465: the body is written as total -> total + price, and the documentation says the left side of -> “receives the accumulated value of the previous iteration”. uniq: true dropped the repeated 300 and 100, and into: MapSet.new() kept the three different remainders.

Comprehensions and Enum pipelines overlap. Answer the two questions about a loop of your own to see which fits:

Bitstring generators walk a binary

A bitstring generatorbitstring generator: A generator written <<pattern <- binary>>, which takes a binary apart piece by piece, such as <<r::8, g::8, b::8 <- pixels>> for three bytes at a time. A last piece too short for the pattern is skipped without an error. takes a binary apart instead of a list, piece by piece, with a binary pattern on the left of <-. The comprehensions guide’s example is pixels, and pixels.exs decodes nine bytes of red, green and blue values into tuples and back:

# An image's pixels as raw bytes, three to a pixel (red, green, blue), turned
# into tuples and back with bitstring generators.
pixels = <<255, 0, 0, 0, 128, 255, 10, 20, 30>>
IO.puts("#{byte_size(pixels)} bytes")

# Each step of the generator takes the next three bytes, one 8-bit number each.
tuples = for <<r::8, g::8, b::8 <- pixels>>, do: {r, g, b}
IO.inspect(tuples, label: "pixels")

# A filter and a calculation on each pixel: the grey value of the bright ones.
grey = for {r, g, b} <- tuples, r + g + b > 100, do: div(r + g + b, 3)
IO.inspect(grey, label: "grey values of the bright pixels", charlists: :as_lists)

# into: <<>> collects binaries, so the tuples become bytes again.
back = for {r, g, b} <- tuples, into: <<>>, do: <<r, g, b>>
IO.inspect(back == pixels, label: "back to the same bytes")

# Two bytes too many: the generator takes the one whole pixel and skips the rest.
IO.inspect(for(<<r::8, g::8, b::8 <- <<255, 0, 0, 7, 8>> >>, do: {r, g, b}), label: "from 5 bytes")

from session import run

run("""
elixir pixels.exs
""")

$ elixir pixels.exs
9 bytes
pixels: [{255, 0, 0}, {0, 128, 255}, {10, 20, 30}]
grey values of the bright pixels: [85, 127]
back to the same bytes: true
from 5 bytes: [{255, 0, 0}]

Each step of <<r::8, g::8, b::8 <- pixels>> took the next three bytes as three 8-bit numbers, so 9 bytes became three tuples. The filter and the body then worked on tuples as in any comprehension, and into: <<>> collected the binaries <<r, g, b>> back into the same 9 bytes. Step through it, then through the binary of 5 bytes:

The last line of the output is the trap: from 5 bytes the generator returned one pixel and skipped the 2 bytes that could not fill the pattern, without an error. A truncated upload or a format with padding at the end of each row gives wrong pictures silently.

Practise: the Game of Life

Conway’s Game of Life is a grid of cells, each alive or dead, that changes in steps: a live cell with 2 or 3 live neighbours stays alive, a dead cell with exactly 3 comes alive, and every other cell is dead in the next step. Here the board is a MapSet of the live cells’ {row, col} positions. Write Life.step/1 and Life.render/3 with comprehensions. Copy the two files into an empty folder:

# Conway's Game of Life on a board of live cells: a MapSet of {row, col} tuples.
# A live cell with 2 or 3 live neighbours stays alive, a dead cell with exactly 3
# comes alive, and every other cell is dead in the next generation. Write both
# functions with for comprehensions; each starter returns a placeholder.
defmodule Life do
  # The live cells of the next generation, as a MapSet.
  def step(live), do: live

  # The board as text: rows lines of cols characters, "#" for a live cell and
  # "." for a dead one, each line ending in "\n".
  def render(_live, _rows, _cols), do: ""
end

Code.require_file("life.ex", __DIR__)
ExUnit.start(seed: 0)

defmodule LifeTest do
  use ExUnit.Case

  @blinker MapSet.new([{1, 0}, {1, 1}, {1, 2}])
  @block MapSet.new([{0, 0}, {0, 1}, {1, 0}, {1, 1}])
  @glider MapSet.new([{0, 1}, {1, 2}, {2, 0}, {2, 1}, {2, 2}])

  test "render draws every row of the board" do
    assert Life.render(@blinker, 3, 3) == "...\n###\n...\n"
    assert Life.render(MapSet.new(), 2, 4) == "....\n....\n"
  end

  test "a lone cell dies, and an empty board stays empty" do
    assert Life.step(MapSet.new([{5, 5}])) == MapSet.new()
    assert Life.step(MapSet.new()) == MapSet.new()
  end

  test "a blinker turns from a row into a column" do
    assert Life.step(@blinker) == MapSet.new([{0, 1}, {1, 1}, {2, 1}])
    assert Life.step(Life.step(@blinker)) == @blinker
  end

  test "a block does not change" do
    assert Life.step(@block) == @block
  end

  test "a glider moves one cell down and one right in four steps" do
    moved = for {r, c} <- @glider, into: MapSet.new(), do: {r + 1, c + 1}
    assert Enum.reduce(1..4, @glider, fn _, live -> Life.step(live) end) == moved
  end
end

Run elixir life_test.exs. The starter returns the board unchanged and an empty drawing, so only the test of a board that does not change passes:

from session import run

# The tests against the starter, whose functions return placeholders.
run("""
cd practice
!elixir life_test.exs
""")

$ cd practice
$ elixir life_test.exs
Running ExUnit with seed: 0, max_cases: 36



  1) test render draws every row of the board (LifeTest)
     life_test.exs:11
     Assertion with == failed
     code:  assert Life.render(@blinker, 3, 3) == "...\n###\n...\n"
     left:  ""
     right: "...\n###\n...\n"
     stacktrace:
       life_test.exs:12: (test)



  2) test a lone cell dies, and an empty board stays empty (LifeTest)
     life_test.exs:16
     Assertion with == failed
     code:  assert Life.step(MapSet.new([{5, 5}])) == MapSet.new()
     left:  MapSet.new([{5, 5}])
     right: MapSet.new([])
     stacktrace:
       life_test.exs:17: (test)



  3) test a blinker turns from a row into a column (LifeTest)
     life_test.exs:21
     Assertion with == failed
     code:  assert Life.step(@blinker) == MapSet.new([{0, 1}, {1, 1}, {2, 1}])
     left:  MapSet.new([{1, 0}, {1, 1}, {1, 2}])
     right: MapSet.new([{0, 1}, {1, 1}, {2, 1}])
     stacktrace:
       life_test.exs:22: (test)

.

  4) test a glider moves one cell down and one right in four steps (LifeTest)
     life_test.exs:30
     Assertion with == failed
     code:  assert Enum.reduce(1..4, @glider, fn _, live -> Life.step(live) end) == moved
     left:  MapSet.new([{0, 1}, {1, 2}, {2, 0}, {2, 1}, {2, 2}])
     right: MapSet.new([{1, 2}, {2, 3}, {3, 1}, {3, 2}, {3, 3}])
     stacktrace:
       life_test.exs:32: (test)


Finished in 0.3 seconds (0.3s on load, 0.00s async, 0.02s sync)

Result: 1/5 passed
Failed: 4 tests
[exit status 2]

Three comprehensions do the work. The eight neighbours of a cell are two generators over -1..1 with a filter that drops {0, 0}. The cells that may be alive next are the live cells and all their neighbours, collected with into: MapSet.new() so each is there once; a filter that counts each one’s live neighbours keeps the cells of the next step. render/3 is a comprehension over the rows with into: "", whose body is a comprehension over the columns, also into: "", followed by "\n". A finished version passes all five tests:

from session import run

# The same tests against a finished Life module.
run("""
cd finished
cp ../practice/life_test.exs .
elixir life_test.exs
""")

$ cd finished
$ cp ../practice/life_test.exs .
$ elixir life_test.exs
Running ExUnit with seed: 0, max_cases: 36

.....
Finished in 0.04 seconds (0.04s on load, 0.00s async, 0.00s sync)

Result: 5 passed

Then try the self-check: it asks about generators, filters and patterns, collecting, and binaries, one question at a time and the idea you know least first, and it explains every answer the moment you give it:

Interview questions

Warm-up: what does a pattern on the left of <- do in an Elixir comprehension?

It filters by shape. Each element of the collection is matched against the pattern; an element that matches binds the pattern’s variables and goes on to the filters and the body, and one that does not match is skipped, with no error. So for {:ok, amount} <- results, do: amount returns the amounts of the successful results only: [1200, 450] from two successes and two errors. That makes it a short way to pick one kind of value out of a list of tagged tuples, and a way to lose errors silently when they should be reported. A function with the same pattern as its only clause raises FunctionClauseError instead, which is the behaviour to choose when a mismatch is a bug.

Two questions from imports and binary data:

Key takeaways

• A comprehension with several generators produces every combination, with the last generator varying fastest; a later generator may use an earlier variable, and an empty generator gives an empty result.
• Filters keep the elements for which they are truthy, and a pattern on the left of <- skips the elements that do not match it, silently, where the same pattern in a function raises FunctionClauseError.
• into: collects into any collectable, such as a map, a string or a MapSet, reduce: threads an accumulator through acc -> … clauses, and uniq: drops repeated results.
• A bitstring generator takes a binary apart piece by piece, <<r::8, g::8, b::8 <- pixels>> three bytes at a time, and skips a last piece that does not fill the pattern, so check the size of the data first.
• Elixir 1.20’s type checker assumes a comprehension’s body runs at least once; Erlang’s comprehensions have a strict generator, <:-, and from Erlang/OTP 29 can emit several results per iteration, which Elixir’s for does not.