Strings, binaries and charlists

byte_size("नमस्ते") is 18, the word has 6 code points, and String.length("नमस्ते") is 3, and all three answers are right: they count bytes, Unicode code points and the characters a reader sees. A program that limits a name to 20 characters, cuts a preview at 100 or compares two names must pick the right one, or it rejects Indian names, splits a letter in half or treats two identical-looking names as different people.

Below, you will measure text three ways, cut it without breaking characters, normalise it before comparing, change its case for the right language, and recognise the charlists that Erlang functions use and the lists of numbers that IEx prints as text. The outputs below are exactly what Elixir 1.20.4 printed, on Erlang/OTP 29.1.1.

A string is UTF-8 bytes, and you can count it three ways

Elixir has one string type, and its guide defines it plainly: “A string is a UTF-8 encoded binary”. A binarybinary: A sequence of bytes, written <<104, 105>>. An Elixir string is a binary whose bytes are valid UTF-8, so "hi" and <<104, 105>> are the same value. is a sequence of bytes, and UTF-8UTF-8: The encoding of Elixir strings. It stores each code point in 1 to 4 bytes: ASCII letters take 1, é takes 2, Devanagari and Tamil letters 3, and most emoji 4. stores the number of each character in one to four bytes. So every string can be counted in three units, and this script counts five strings in all three, writing each non-ASCII character as an escape such as \u0928 so the source shows what is in it:

# Five strings measured three ways. The \u escapes spell out each code point, so nothing is hidden in the source.
words = [
  {"cafe", "cafe"},
  {"e acute", "\u00E9"},
  {"flag of India", "\u{1F1EE}\u{1F1F3}"},
  {"namaste", "\u0928\u092E\u0938\u094D\u0924\u0947"},
  {"vanakkam", "\u0BB5\u0BA3\u0B95\u0BCD\u0B95\u0BAE\u0BCD"}
]

IO.puts("string          bytes  code points  graphemes  text")

for {name, text} <- words do
  counts = [byte_size(text), length(String.codepoints(text)), String.length(text)]
  [bytes, points, graphemes] = Enum.map(counts, &Integer.to_string/1)

  IO.puts(
    String.pad_trailing(name, 14) <>
      String.pad_leading(bytes, 7) <>
      String.pad_leading(points, 13) <>
      String.pad_leading(graphemes, 11) <> "  " <> text
  )
end

# The code points of namaste, each with the UTF-8 bytes that store it.
namaste = "\u0928\u092E\u0938\u094D\u0924\u0947"

for point <- String.codepoints(namaste) do
  [number] = String.to_charlist(point)
  bytes = for byte <- :binary.bin_to_list(point), do: Integer.to_string(byte, 16)
  IO.puts("U+" <> String.pad_leading(Integer.to_string(number, 16), 4, "0") <> "  " <> Enum.join(bytes, " "))
end

IO.inspect(String.graphemes(namaste), label: "graphemes of namaste")

# The graphemes of vanakkam: where String.length/1 found its 4.
vanakkam = "\u0BB5\u0BA3\u0B95\u0BCD\u0B95\u0BAE\u0BCD"
IO.inspect(String.graphemes(vanakkam), label: "graphemes of vanakkam")

from session import run

run("""
elixir text_units.exs
""")

$ elixir text_units.exs
string          bytes  code points  graphemes  text
cafe                4            4          4  cafe
e acute             2            1          1  é
flag of India       8            2          1  🇮🇳
namaste            18            6          3  नमस्ते
vanakkam           21            7          4  வணக்கம்
U+0928  E0 A4 A8
U+092E  E0 A4 AE
U+0938  E0 A4 B8
U+094D  E0 A5 8D
U+0924  E0 A4 A4
U+0947  E0 A5 87
graphemes of namaste: ["न", "म", "स्ते"]
graphemes of vanakkam: ["வ", "ண", "க்க", "ம்"]

• byte_size/1 counts bytes: 4 for “cafe”, 2 for é, 18 for नमस्ते. That is what the text costs in memory, in a file or on the network, and the documentation says it “happens in constant time”.
• String.codepoints/1 lists the code pointscode point: The number that Unicode gives to a character or a mark, written U+0928 for न. String.codepoints/1 lists the code points of a string., the numbers Unicode gives to characters and marks. The flag of India is two of them, two regional indicator letters.
• String.length/1 counts graphemesgrapheme: What a reader sees as one character, made of one or more code points: स्ते in नमस्ते is one grapheme of four code points. String.length/1 counts graphemes., what a reader sees as one character. The flag is one grapheme, and so is स्ते, though it holds four code points.

Unicode’s text segmentation annex treats grapheme clusters as the characters a user perceives, and that is why String.length/1, String.slice/3 and String.reverse/1 work in graphemes: they treat text the way a reader sees it. The String module follows the Unicode Standard 17.0.0. Step through the Devanagari word from its bytes to its graphemes:

The Tamil word வணக்கம் is 21 bytes and 7 code points, and String.length/1 gives 4. The last line of the program shows the four: Elixir keeps க, the virama ் and the next க together in one grapheme, as it kept स्त, and ம் with its virama in another. Tamil writes க் and க as two letters, though, and Unicode’s list of named sequences even names க் on its own, TAMIL CONSONANT K, so a Tamil reader counts five letters where Elixir counts four graphemes. Unicode’s segmentation annex calls graphemes a best-effort approximation of what readers see, so test a length limit with names in the scripts your users write.

Cutting a string by bytes can split a character in half

Because a string is a binary, the byte functions of Kernel work on it too, and they do not know where one character ends and the next begins. Watch what happens to é in “héllo”:

from session import shell

# Strings are binaries: counting, cutting and joining bytes.
shell([
    'byte_size("héllo")',
    'String.length("héllo")',
    'binary_part("héllo", 0, 2)',
    'String.valid?(binary_part("héllo", 0, 2))',
    'String.slice("héllo", 0, 2)',
    "<<104, 105>>",
    'String.valid?(<<0xFF>>)',
    '"tea for " <> "two"',
    '"#{3} cups of #{:tea}"',
    "System.halt()",
])

$ iex
Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:18:18] [ds:18:18:10] [async-threads:1] [jit]

Interactive Elixir (1.20.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> byte_size("héllo")
6
iex(2)> String.length("héllo")
5
iex(3)> binary_part("héllo", 0, 2)
<<104, 195>>
iex(4)> String.valid?(binary_part("héllo", 0, 2))
false
iex(5)> String.slice("héllo", 0, 2)
"hé"
iex(6)> <<104, 105>>
"hi"
iex(7)> String.valid?(<<0xFF>>)
false
iex(8)> "tea for " <> "two"
"tea for two"
iex(9)> "#{3} cups of #{:tea}"
"3 cups of tea"
iex(10)> System.halt()

byte_size("héllo") is 6 while String.length("héllo") is 5, since é takes two bytes. binary_part("héllo", 0, 2) took the first two bytes and returned <<104, 195>>: the h and the first half of é. IEx printed it byte by byte, in the <<>> form, rather than as text, because it is no longer valid UTF-8, and String.valid?/1 said so. String.slice("héllo", 0, 2) counted graphemes instead and returned “hé”.

The <<>> syntax writes a binary byte by byte. <<104, 105>> is the same value as “hi”, which is why IEx printed it as a string, and <<0xFF>> is a binary but not a string, because 0xFF never appears in UTF-8. Two ways build text from parts: <> joins two binaries, and interpolation, "#{3} cups of #{:tea}", turns each value into text first.

Text that looks the same can differ: normalise before comparing

The letter é can be written in two ways: as one code point, U+00E9, or as e followed by U+0301, a combining acute accent. A keyboard usually types the first; text copied from some file names, documents or other systems may hold the second. They look identical and compare as different:

from session import shell

# Two spellings of é: one code point, or e followed by a combining accent; and a look-alike that is another letter.
shell([
    'composed = "\\u00E9"',
    'decomposed = "e\\u0301"',
    "composed == decomposed",
    "{byte_size(composed), byte_size(decomposed)}",
    "{String.length(composed), String.length(decomposed)}",
    "String.normalize(decomposed, :nfc) == composed",
    "String.equivalent?(composed, decomposed)",
    'String.normalize("\\u0430", :nfc) == "a"',
    "System.halt()",
])

$ iex
Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:18:18] [ds:18:18:10] [async-threads:1] [jit]

Interactive Elixir (1.20.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> composed = "\u00E9"
"é"
iex(2)> decomposed = "e\u0301"
"é"
iex(3)> composed == decomposed
false
iex(4)> {byte_size(composed), byte_size(decomposed)}
{2, 3}
iex(5)> {String.length(composed), String.length(decomposed)}
{1, 1}
iex(6)> String.normalize(decomposed, :nfc) == composed
true
iex(7)> String.equivalent?(composed, decomposed)
true
iex(8)> String.normalize("\u0430", :nfc) == "a"
false
iex(9)> System.halt()

The composed é is 2 bytes and the decomposed one 3, and == compares bytes, so it returned false, though both are one grapheme. Normalisationnormalisation (Unicode): Rewriting text into one standard form so that text that looks the same has the same bytes. NFC composes e and a combining accent into the single code point é. rewrites text into one standard form. With :nfc, String.normalize/2 composes e and the accent into U+00E9, and the comparison came out true. String.equivalent?/2 answers the same question directly; its documentation says it normalises both strings each time it is called, so to compare many strings, normalise each one once and compare the results.

Unicode’s normalisation annex defines the forms. NFC, which composes, is the usual choice for text you store and compare; NFD decomposes. NFKC and NFKD also fold compatibility characters such as ligatures, and the String.normalize/2 documentation warns that they “should not be blindly applied to arbitrary text”. Normalising unifies only the spellings that Unicode counts as the same character: the last line compared the Cyrillic letter а, U+0430, with the Latin a, and they stayed different, however alike they look.

Upper and lower case depend on the language

Changing case is not one letter in, one letter out. German, Greek and Turkish each need something different, and String.upcase/2 and String.downcase/2 take a mode for that:

from session import shell

# Changing case follows the language: German sharp s, Greek final sigma and the Turkish dotted i.
shell([
    'String.upcase("straße")',
    'String.downcase("ΟΔΟΣ")',
    'String.downcase("ΟΔΟΣ", :greek)',
    'String.upcase("istanbul")',
    'String.upcase("istanbul", :turkic)',
    'String.upcase("नमस्ते") == "नमस्ते"',
    "System.halt()",
])

$ iex
Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:18:18] [ds:18:18:10] [async-threads:1] [jit]

Interactive Elixir (1.20.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> String.upcase("straße")
"STRASSE"
iex(2)> String.downcase("ΟΔΟΣ")
"οδοσ"
iex(3)> String.downcase("ΟΔΟΣ", :greek)
"οδος"
iex(4)> String.upcase("istanbul")
"ISTANBUL"
iex(5)> String.upcase("istanbul", :turkic)
"İSTANBUL"
iex(6)> String.upcase("नमस्ते") == "नमस्ते"
true
iex(7)> System.halt()

String.upcase("straße") returned “STRASSE”: Unicode’s special casing rules give the sharp s the two letters SS as its upper case, so one character became two, and the string grew. In Greek a lower-case sigma is written ς at the end of a word and σ elsewhere. The default mode wrote “οδοσ”; the :greek mode, which “includes the context sensitive mappings found in Greek” in the documentation’s words, wrote the correct “οδος”. In Turkish, i and ı are two letters, and the upper case of i is İ: the default mode gave “ISTANBUL”, the :turkic mode “İSTANBUL”. Devanagari has no case, so String.upcase/1 left नमस्ते as it was.

Two consequences follow. A length checked before changing case may not hold after it, as “straße” showed. And comparing two texts without regard to case means changing both to the same case with the same mode, after normalising them.

Charlists are lists of code points, and some lists print as text

In Erlang’s own source code, a string in double quotes is not a type of its own but shorthand for a list of character codes, and many Erlang functions take or return text in that form. Elixir calls such a list a charlistcharlist: A list of code points, such as ~c"hi", which is the list [104, 105]. Many Erlang functions take and return text as charlists rather than as strings. and writes it with the ~c sigil: ~c"hi" is the list [104, 105]. Here is where charlists turn up, and the surprise they cause:

from session import shell

# Charlists: lists of code points, and lists of numbers that IEx prints as text.
shell([
    'is_list(~c"hi")',
    "[104, 105]",
    "Enum.chunk_every(1..7, 3)",
    "IO.inspect([9, 36, 81], charlists: :as_lists)",
    'String.to_charlist("hé")',
    "List.to_string([104, 233])",
    ":erlang.integer_to_list(42)",
    '~c"tea" ++ ~c"pot"',
    "System.halt()",
])

$ iex
Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:18:18] [ds:18:18:10] [async-threads:1] [jit]

Interactive Elixir (1.20.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> is_list(~c"hi")
true
iex(2)> [104, 105]
~c"hi"
iex(3)> Enum.chunk_every(1..7, 3)
[[1, 2, 3], [4, 5, 6], ~c"\a"]
iex(4)> IO.inspect([9, 36, 81], charlists: :as_lists)
[9, 36, 81]
~c"\t$Q"
iex(5)> String.to_charlist("hé")
[104, 233]
iex(6)> List.to_string([104, 233])
"hé"
iex(7)> :erlang.integer_to_list(42)
~c"42"
iex(8)> ~c"tea" ++ ~c"pot"
~c"teapot"
iex(9)> System.halt()

~c"hi" is a list, and the list [104, 105] came back as ~c"hi". IEx prints any list whose numbers are all printable ASCII codes as a charlist, and the List documentation counts some control codes as printable too, such as \a, 7, and \t, 9. So Enum.chunk_every(1..7, 3) showed its last chunk, [7], as ~c"\a". The numbers are still there; only the display changed. To see them as numbers, pass charlists: :as_lists to IO.inspect/2 or inspect/2, as the fourth line did: it printed [9, 36, 81], and IEx then showed the same value as ~c"\t$Q".

To cross between the two forms, String.to_charlist/1 turns a string into a charlist and List.to_string/1 turns a charlist back; [104, 233] holds é, which is not ASCII, so IEx printed it as numbers. Erlang functions such as :erlang.integer_to_list/1 return charlists, here ~c"42". And charlists are joined with ++, like any list, never with <>. Those two conversion functions are also where the security fix of 1.20.4 lies:

Practise: initials in any script

Write Initials.from/1, which returns the initials of a name: the first grapheme of each word, in upper case, after normalising the name to NFC. “asha rao” gives “AR”, and the Devanagari आशा राव gives आरा, because रा, the letter र with its vowel sign, is the first grapheme of राव. Copy the two files into an empty folder:

defmodule Initials do
  @moduledoc "Initials of a name in any script: the first grapheme of each word, in upper case."

  @doc """
  Returns the initials of `name`: the first grapheme of every word, upper-cased, with the text
  normalised to NFC first. Words are separated by any run of spaces.
  """
  def from(_name) do
    # Drop the underscore once you use the argument.
    ""
  end
end

Code.require_file("initials.ex", __DIR__)
ExUnit.start(seed: 0)

defmodule InitialsTest do
  use ExUnit.Case

  test "two Latin words" do
    assert Initials.from("asha rao") == "AR"
  end

  test "an accented first letter, typed as e and a combining accent" do
    # "e\u0301" is e followed by U+0301; the expected É is the single code point U+00C9.
    assert Initials.from("e\u0301mile zola") == "\u00C9Z"
  end

  test "Devanagari: the first grapheme of each word, which may be several code points" do
    assert Initials.from("\u0906\u0936\u093E \u0930\u093E\u0935") == "\u0906\u0930\u093E"
  end

  test "extra spaces between and around the words" do
    assert Initials.from("  kiran   desai ") == "KD"
  end
end

Run elixir initials_test.exs. The starter returns “” for every name, so all four tests fail:

from session import run

# The tests against the starter, which returns "" for every name.
run("""
cd practice
!elixir initials_test.exs
""")

$ cd practice
$ elixir initials_test.exs
Running ExUnit with seed: 0, max_cases: 36



  1) test two Latin words (InitialsTest)
     initials_test.exs:7
     Assertion with == failed
     code:  assert Initials.from("asha rao") == "AR"
     left:  ""
     right: "AR"
     stacktrace:
       initials_test.exs:8: (test)



  2) test an accented first letter, typed as e and a combining accent (InitialsTest)
     initials_test.exs:11
     Assertion with == failed
     code:  assert Initials.from("émile zola") == "ÉZ"
     left:  ""
     right: "ÉZ"
     stacktrace:
       initials_test.exs:13: (test)



  3) test Devanagari: the first grapheme of each word, which may be several code points (InitialsTest)
     initials_test.exs:16
     Assertion with == failed
     code:  assert Initials.from("आशा राव") == "आरा"
     left:  ""
     right: "आरा"
     stacktrace:
       initials_test.exs:17: (test)



  4) test extra spaces between and around the words (InitialsTest)
     initials_test.exs:20
     Assertion with == failed
     code:  assert Initials.from("  kiran   desai ") == "KD"
     left:  ""
     right: "KD"
     stacktrace:
       initials_test.exs:21: (test)


Finished in 0.02 seconds (0.02s on load, 0.00s async, 0.00s sync)

Result: 0/4 passed
Failed: 4 tests
[exit status 2]

String.normalize/2 comes first, String.split/1 splits at any run of spaces and drops the empty words, and String.first/1 returns the first grapheme of a word, so a vowel sign stays with its letter. Then Enum.map/2 takes the first grapheme of every word, Enum.join/1 joins them, and String.upcase/1 changes the case. Leave out the normalising, and the second test fails, with a hint from ExUnit that names the cause:

from session import run

# The tests against a version that skips String.normalize/2.
run("""
cd no_nfc
cp ../practice/initials_test.exs .
!elixir initials_test.exs
""")

$ cd no_nfc
$ cp ../practice/initials_test.exs .
$ elixir initials_test.exs
Running ExUnit with seed: 0, max_cases: 36

.

  1) test an accented first letter, typed as e and a combining accent (InitialsTest)
     initials_test.exs:11
     Assertion with == failed
     code:  assert Initials.from("émile zola") == "ÉZ"
     left:  "ÉZ"
     right: "ÉZ"
     hint:  you are comparing strings that have the same visual representation but are made of different Unicode codepoints
     stacktrace:
       initials_test.exs:13: (test)

..
Finished in 0.01 seconds (0.01s on load, 0.00s async, 0.00s sync)

Result: 3/4 passed
Failed: 1 test
[exit status 2]

Both sides of that failure print as “ÉZ”, and the hint says that the two strings “have the same visual representation but are made of different Unicode codepoints”. With String.normalize(name, :nfc) first, all four tests pass:

from session import run

# The same tests against a finished from/1.
run("""
cd finished
cp ../practice/initials_test.exs .
elixir initials_test.exs
""")

$ cd finished
$ cp ../practice/initials_test.exs .
$ elixir initials_test.exs
Running ExUnit with seed: 0, max_cases: 36

....
Finished in 0.01 seconds (0.01s on load, 0.00s async, 0.00s sync)

Result: 4 passed

Now the self-check for this lesson. Each question arrives on its own, the weakest of your five topics goes first, and the reason behind every answer appears the moment you pick one:

Interview questions

Warm-up: what is the difference between byte_size/1 and String.length/1 in Elixir?

byte_size/1 counts the bytes of a binary, and it runs in constant time because the VM stores the size; for a string it is the size of its UTF-8 encoding. String.length/1 counts graphemes, the characters a reader sees, and it has to read the whole string to find where each grapheme ends. They agree for ASCII text, where every character is one byte, and differ everywhere else: “héllo” is 6 bytes and 5 graphemes, and the Hindi greeting नमस्ते is 18 bytes, 6 code points and 3 graphemes. Use byte_size/1 for limits of storage and transport, such as a database column or a packet, and String.length/1 for limits people see, such as the length of a name on a form.

Two questions from forms and logins that handle names in many scripts:

Key takeaways

• A string is a UTF-8 binary: byte_size/1 counts bytes, String.codepoints/1 lists code points and String.length/1 counts graphemes, and नमस्ते is 18, 6 and 3 of them.
• Byte functions such as binary_part/3 can split a character, as <<104, 195>> showed; cut text with String.slice/3.
• Text that looks the same can hold different code points, so normalise to NFC with String.normalize/2 where text enters the system, before comparing it or using it as a key.
• Casing depends on the language: “straße” upper-cases to “STRASSE”, Greek needs the :greek mode and Turkish the :turkic mode.
• A charlist is a list of code points used by Erlang functions; IEx prints any list of printable ASCII codes as one, so use charlists: :as_lists to see numbers, and run Elixir 1.20.4 or another patched release for the List.to_string/1 fix.