Strings and Unicode in R

nchar("नमस्ते") is 6, yet a reader sees three letters, the word takes 18 bytes in a file and 4 columns on screen, and sprintf("%-10s|", "नमस्ते") pads it with no spaces at all. Each of those numbers is right; they count different things. Text in R is easy to build and print until it leaves plain English, and every function you use counts characters, bytes or columns.

On recorded runs of R 4.6.1 you will build text with paste() and sprintf(), format numbers with format() and formatC(), measure and slice strings in Devanagari and Tamil without cutting a letter in two, read text whose encoding is not UTF-8, and write a function that groups the digits of an amount the Indian way.

paste() and sprintf() build text from vectors

paste("ward", c("A", "B"), sep = "_") returns two strings, "ward_A" and "ward_B": like most of R, the string functions work element by element. paste() joins its arguments with sep, a space unless you say otherwise, and paste0() joins them with nothing; collapse then joins the results into one string. sprintf() fills a template, and its help page says it recycles every argument, the template included, to the length of the longest. The codes used most:

The run tries each, with the edge cases that catch people out:

from session import run, version

version()
# R reads build_strings.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f build_strings.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f build_strings.R
> # paste() joins element by element; collapse joins the results into one string.
> paste("ward", c("A", "B"), sep = "_")
[1] "ward_A" "ward_B"
> paste(c("a", "b"), collapse = "+")
[1] "a+b"
> paste0("id-", 1:3)
[1] "id-1" "id-2" "id-3"
> # Edge cases: nothing to join gives "", NULL is dropped, NA becomes the text "NA".
> paste(character(0), collapse = "+")
[1] ""
> paste("total:", NULL)
[1] "total: "
> paste("weight", NA)
[1] "weight NA"
> # sprintf() fills a template, element by element, recycling its arguments.
> sprintf("%s scored %d", c("Asha", "Ravi"), c(91L, 78L))
[1] "Asha scored 91" "Ravi scored 78"
> sprintf("%5.1f|%-6s|%04d|%%", 3.14159, "ab", 42L)
[1] "  3.1|ab    |0042|%"
> tryCatch(sprintf("%d", 3.5), error = conditionMessage)
[1] "invalid format '%d'; use format %f, %e, %g or %a for numeric objects"
> # format() and formatC() turn numbers into text with set decimals, widths and marks.
> format(2, nsmall = 2)
[1] "2.00"
> format(1234567.891, big.mark = ",", nsmall = 2)
[1] "1,234,567.89"
> formatC(c(1, 1234567.891), format = "f", digits = 2, big.mark = ",")
[1] "1.00"         "1,234,567.89"
> formatC(42, width = 8, flag = "0")
[1] "00000042"
> format(c(1, 10, 100))
[1] "  1" " 10" "100"

• Edge cases of paste(). Nothing to collapse gave "", a NULL argument was dropped, leaving the trailing space of "total: ", and NA became the two letters "NA", so paste("weight", NA) reads like a value.
• %d wants a whole number. sprintf("%d", 3.5) stopped with “invalid format ‘%d’; use format %f, %e, %g or %a for numeric objects”; a whole number stored as a double, such as 3, is accepted.
• format() and formatC() for numbers. format(2, nsmall = 2) gave "2.00", big.mark = "," grouped the thousands of 1234567.891, and formatC() set the decimals with format = "f" and digits, and padded 42 with zeros to 8 places. format() of several numbers pads them to one common width, " 1", " 10", "100", so they line up in a column.

A string has characters, bytes and a display width

The help page of nchar() gives three answers to “how long is this string?”: type = "chars", the default, counts characters; "bytes" counts the bytes needed to store it; and "width" counts the columns it takes when printed in a monospaced font, where combining characters count zero. In English the three agree. In Hindi they do not, and a fourth count, the letters a reader sees, differs again. Step through the word:

Each box is a code pointcode point: The number Unicode gives one character, written like U+0928 for the Devanagari letter na. R's nchar() counts code points, and utf8ToInt() returns them., one character in Unicode’s sense, and nchar() counts those. What a reader calls a letter is a grapheme clustergrapheme cluster: What a reader sees as one letter, which may be several code points: a consonant with a vowel sign, or a conjunct of two consonants joined by a virama. stringi counts them., which Unicode’s rules for text segmentation define and which the stringi package counts. The run measures the word, slices it, reverses it, and pads it:

from session import run, version

version()
# R reads devanagari.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f devanagari.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f devanagari.R
> # One Hindi word: how many characters?
> x <- "नमस्ते"
> nchar(x)
[1] 6
> nchar(x, type = "bytes")
[1] 18
> nchar(x, type = "width")
[1] 4
> # nchar() counts code points; here are the six, as Unicode numbers.
> sprintf("U+%04X", utf8ToInt(x))
[1] "U+0928" "U+092E" "U+0938" "U+094D" "U+0924" "U+0947"
> # Slicing by code points can cut a letter in two.
> substr(x, 1, 3)
[1] "नमस"
> # Reversing the code points leaves a vowel sign with nothing to sit on.
> paste(rev(strsplit(x, "")[[1]]), collapse = "")
[1] "ेत्समन"
> # stringi's stri_reverse() reverses code points too, with the same result.
> stringi::stri_reverse(x)
[1] "ेत्समन"
> # stringi counts what a reader sees as letters: grapheme clusters.
> stringi::stri_count_boundaries(x, type = "character")
[1] 3
> stringi::stri_split_boundaries(x, type = "character")[[1]]
[1] "न"  "म"  "स्ते"
> # A Tamil word: five code points, three letters, four columns.
> t <- "தமிழ்"
> c(chars = nchar(t), bytes = nchar(t, type = "bytes"), width = nchar(t, type = "width"),
+   letters = stringi::stri_count_boundaries(t, type = "character"))
  chars   bytes   width letters 
      5      15       4       3 
> # Padding: sprintf() counts bytes, formatC() and format() count display columns.
> sprintf("%-10s|", c("tea", x))
[1] "tea       |" "नमस्ते|"      
> paste0(formatC(c("tea", x), width = -10), "|")
[1] "tea       |" "नमस्ते      |"
> paste0(format(c("tea", x), width = 10), "|")
[1] "tea       |" "नमस्ते      |"

• Counting. Six code points, 18 bytes, 4 columns and 3 letters for the Hindi word; 5, 15, 4 and 3 for the Tamil one. Count with stringi::stri_count_boundaries(x, type = "character") when the answer must match what a reader sees, for example to check the length of a name on a form.
• Slicing and reversing. substr(x, 1, 3) cut the conjunct in two, and reversing the code points, by hand or with stringi’s stri_reverse(), left the vowel sign at the front with nothing to sit on. Split into letters first, with stri_split_boundaries(), and slice or reverse those.
• Padding. sprintf("%-10s|", ...) padded "tea" to 10 but added no space to the Hindi word, because its help page says that widths of %s are counted in bytes, and 18 bytes are already more than 10. formatC() and format() padded both to 10 columns.

Slicing and case follow code points and the platform

substring("PATIENT-0042", 9, NULL) returns "0042": R 4.6.0 lets substr() and substring() take NULL as the end, meaning to the end of the string, as R’s news for that release says. Positions in both count code points, so they are safe for text in which every character stands alone, such as an ID, and unsafe in the middle of an Indian word, as the last section showed. strsplit() cuts strings at a separator and returns a list, one element per string, and trimws() and chartr() remove spaces at the ends and replace characters one for one. Case is where platforms differ:

from session import run, version

version()
# R reads transform.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f transform.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f transform.R
> # substr() and substring() slice by position; since R 4.6.0 NULL means "to the end".
> substr("PATIENT-0042", 9, 12)
[1] "0042"
> substring("PATIENT-0042", 9, NULL)
[1] "0042"
> substring("abcdef", 1:3, 3:5)
[1] "abc" "bcd" "cde"
> # strsplit() returns a list, one element per input string.
> strsplit(c("a,b,,c", "d"), ",")
[[1]]
[1] "a" "b" ""  "c"

[[2]]
[1] "d"

> # Trimming and character-for-character replacement.
> trimws("  padded  ")
[1] "padded"
> chartr("abc", "xyz", "aabbcc")
[1] "xxyyzz"
> # Case mapping in base R depends on the platform; stringr uses ICU everywhere.
> toupper("straße")
[1] "STRAßE"
> stringr::str_to_upper("straße")
[1] "STRASSE"
> tolower("ÉCOLE")
[1] "école"
> # ICU also knows language rules: the Turkish dotted capital I.
> stringr::str_to_lower("İSTANBUL", locale = "tr")
[1] "istanbul"
> stringr::str_to_lower("İSTANBUL", locale = "en")
[1] "i̇stanbul"

The help page of toupper() says these functions are platform-dependent, usually using the operating system’s services, so the same line can give different results on Windows, macOS and Linux. stringr’s case functions go through the ICU library everywhere, and its locale argument applies language rules: in English, lowering the Turkish capital İ kept its dot as a separate mark, "i̇stanbul".

An encoding says how the bytes become text

A file holds bytes, not letters: the word café saved by an old system in Latin-1 is the four bytes 63 61 66 E9, and the same word in UTF-8 is five, 63 61 66 C3 A9. A character encodingcharacter encoding: The rule that turns a file's bytes into characters. UTF-8 is the usual one; a file saved as UTF-16LE or Latin-1 reads as garbled text or fails unless its encoding is named when it is read. More in “Reading and writing CSV and text data” is the rule that turns one into the other, and the bytes do not say which rule was used. Follow one file, read twice:

from session import run, version

version()
# R reads encodings.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f encodings.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f encodings.R
> # This R runs in a UTF-8 locale, as almost all installations now do.
> l10n_info()$`UTF-8`
[1] TRUE
> # Strings typed in a UTF-8 script are marked UTF-8; plain ASCII is never marked.
> Encoding(c("abc", "café"))
[1] "unknown" "UTF-8"  
> # The same word saved by an old system in Latin-1: the é is one byte, E9.
> latin <- rawToChar(as.raw(c(0x63, 0x61, 0x66, 0xe9)))
> latin
[1] "caf\xe9"
> validUTF8(latin)
[1] FALSE
> # iconv() converts the bytes from Latin-1 to UTF-8, where é takes two bytes.
> fixed <- iconv(latin, from = "latin1", to = "UTF-8")
> fixed
[1] "café"
> charToRaw(fixed)
[1] 63 61 66 c3 a9
> # Reading a Latin-1 file: say its encoding when you open it.
> writeBin(as.raw(c(0x63, 0x61, 0x66, 0xe9, 0x0a)), "names-latin1.txt")
> readLines("names-latin1.txt")
[1] "caf\xe9"
> readLines(file("names-latin1.txt", encoding = "latin1"))
[1] "café"
> # Text in any script can be written with \u escapes, so a script stays plain ASCII.
> "\u0928\u092e\u0938\u094d\u0924\u0947"
[1] "नमस्ते"
> # Raw strings (R 4.0.0 and later) keep backslashes as they are.
> r"(C:\Users\data)"
[1] "C:\\Users\\data"
> writeLines(r"(C:\Users\data)")
C:\Users\data

• The session’s encoding. l10n_info() reported a UTF-8 session. R’s news for 4.2.0 says R uses UTF-8 as the native encoding on recent Windows systems too, so UTF-8 is now the common case everywhere.
• What R records. Encoding() marked the typed "café" as UTF-8 and left the ASCII "abc" as "unknown", which is harmless, since ASCII is the same in every common encoding.
• Fixing bytes. The Latin-1 bytes printed as "caf\xe9" and failed validUTF8(). Converting them with iconv(from = "latin1", to = "UTF-8"), or opening the file with file(..., encoding = "latin1"), gave "café". The fix belongs where the text enters R: the reader functions take an encoding argument for that.
• Writing text in any script. "\u0928" is the Devanagari letter na written in plain ASCII, so a script can hold Hindi without depending on its own encoding. A raw string, r"(C:\Users\data)", keeps backslashes as typed, as R 4.0.0 allowed.

Indian digit grouping needs a small function

format(123456789.5, big.mark = ",", nsmall = 2) gives "123,456,789.50": R groups digits in threes, at one fixed interval. The Indian system groups the last three digits and then every two, 12,34,56,789.50, the pattern #,##,##0.### that Unicode’s locale data gives English and Hindi in India, and neither format() nor formatC() can produce that. The function in the run builds it from this lesson’s tools.

Step through the function on one amount:

from session import run, version

version()
# R reads format_inr.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f format_inr.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f format_inr.R
> # Indian digit grouping: the last three digits, then groups of two.
> group_indian <- function(digits) {
+   n <- nchar(digits)
+   if (n <= 3) return(digits)
+   head <- substr(digits, 1, n - 3)
+   ends <- rev(seq(nchar(head), 1, by = -2))
+   pairs <- substring(head, pmax(ends - 1, 1), ends)
+   paste0(paste(pairs, collapse = ","), ",", substr(digits, n - 2, n))
+ }
> format_inr <- function(x, paise = TRUE) {
+   stopifnot(is.numeric(x))
+   one <- function(v) {
+     if (is.na(v)) return(NA_character_)
+     text <- sprintf(if (paise) "%.2f" else "%.0f", abs(v))
+     parts <- strsplit(text, ".", fixed = TRUE)[[1]]
+     rupees <- group_indian(parts[1])
+     amount <- if (paise) paste0(rupees, ".", parts[2]) else rupees
+     # The sign comes from the rounded text, so -0.004 is not written as -0.00.
+     negative <- v < 0 && as.numeric(text) > 0
+     paste0(if (negative) "-" else "", "\u20b9", amount)
+   }
+   vapply(x, one, character(1))
+ }
> amounts <- c(999, 100000, 123456789.5, -2500, 0, NA)
> # Base R groups in threes only.
> format(amounts, big.mark = ",", nsmall = 2)
[1] "        999.00" "    100,000.00" "123,456,789.50" "     -2,500.00"
[5] "          0.00" "            NA"
> format_inr(amounts)
[1] "₹999.00"          "₹1,00,000.00"     "₹12,34,56,789.50" "-₹2,500.00"      
[5] "₹0.00"            NA                
> format_inr(c(1234567, 99.4), paise = FALSE)
[1] "₹12,34,567" "₹99"       
> # Rounding at the edges: a tiny negative amount, and one that rounds up a digit.
> format_inr(c(-0.004, 99.995))
[1] "₹0.00"   "₹100.00"
> # A check of the grouping on the lengths where it changes.
> vapply(c("999", "1000", "12345", "123456", "1234567"), group_indian, character(1))
        999        1000       12345      123456     1234567 
      "999"     "1,000"    "12,345"  "1,23,456" "12,34,567" 

group_indian() finds the end of each pair of digits with seq() counting down by 2 from the end, and substring() cuts all the pairs at once, a vectorised use of the slicing shown earlier. format_inr() then works on one amount at a time inside vapply(), with a character(1) template, so a missing amount stays NA and the result always has one string per amount. One line tests the edges of rounding: -0.004 rounds to 0.00, and the function decides the sign from the rounded text, so it does not print a minus zero, while 99.995 rounds up into a new digit, 100.00. The last line checks the lengths at which the grouping changes, from three digits to seven: test a formatting function at exactly those boundaries.

Practise: predict, then check

Work out each value first, counting characters, bytes or letters as each line asks.

from session import run, version

version()
# R reads practice.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f practice.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f practice.R
> # 1. paste() and collapse:
> paste(c("x", "y"), 1:2, sep = "-", collapse = " ")
[1] "x-1 y-2"
> # 2. A template:
> sprintf("%05.1f", 3.14159)
[1] "003.1"
> # 3. Counting a word in Devanagari:
> nchar("भारत")
[1] 4
> stringi::stri_count_boundaries("भारत", type = "character")
[1] 3
> # 4. The end of a string:
> substring("2026-27", 6, NULL)
[1] "27"

sep joined each pair and collapse the pairs; %05.1f padded 3.1 with zeros to five characters; "भारत" has four code points and three letters, because the vowel sign joins the first; and the substring from position 6 to the end is "27".

Next, the questions below. Each round begins with the idea you have found hardest so far, and every choice explains which count, rule or function it rests on:

Interview questions

Warm-up: what is the difference between nchar(x), nchar(x, type = “bytes”) and nchar(x, type = “width”)?

nchar(x) counts characters in Unicode’s sense, code points, so "नमस्ते" has 6. type = "bytes" counts the bytes the string takes in its encoding: 18 in UTF-8, where each Devanagari code point needs three bytes and an ASCII letter one. type = "width" counts the columns the string takes in a monospaced font, where marks that take no space of their own, such as the virama and the vowel sign e, count zero and wide East Asian characters two, so the same word is 4 columns wide. None of them counts the letters a reader sees, 3 here, which needs grapheme rules such as stringi::stri_count_boundaries(). The difference matters for limits on a form, for padding columns, since sprintf() counts bytes, and for slicing text.

Key takeaways

• paste() and sprintf() work element by element and recycle their arguments; collapse joins the results, NA becomes the text "NA", and %d accepts only whole numbers.
• A string has a count of code points, of bytes and of display columns, and a reader’s letters are grapheme clusters, which stringi counts; in Indian scripts all four can differ.
• substr() and substring() slice by code points and take NULL as the end since R 4.6.0; base R’s case mapping depends on the platform, while stringr’s uses ICU everywhere.
• Bytes do not say their encoding, so declare it when you read, convert to UTF-8 once with iconv() or the reader’s encoding argument, and write UTF-8.
• R groups digits in threes only, so the Indian grouping needs a small function, tested at the lengths where the grouping changes.