Lists and nested data

A clinic’s system exports each patient as one record: an id, an age, three readings and a list of visits, each visit with its own reason and fee. No vector can hold that, because a vector has one type and one level. A list can: it holds any objects, other lists included, and a model fitted in R comes back as one too.

On runs of R 4.6.1 you will build and read nested lists, choose between [ ], [[ ]] and $ without the partial matching that makes l$al return l$alpha, add and delete elements, and flatten a list of records into a data frame with one row per record.

A list holds objects of any type, and str() shows its shape

list(id = "P-104", age = 52L, ...) makes a listlist: R's container for objects of any type and length, each element with an optional name. [ ] returns a smaller list, while [[ ]] and $ return one element itself.: an ordered set of elements, each of which can be any object, here a string, an integer, a named numeric vector and another list. An Introduction to R puts it plainly: the components need not be of the same mode or type. Step through the record’s four elements:

The run builds the record and prints its shape:

from session import run, version

version()
# R reads record.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f record.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f record.R
> # One patient's record from a clinic system.
> rec <- list(
+   id = "P-104",
+   age = 52L,
+   vitals = c(sbp = 138, dbp = 86, pulse = 74),
+   visits = list(
+     list(reason = "check-up", fee = 300),
+     list(reason = "fever", fee = 450)
+   )
+ )
> length(rec)
[1] 4
> names(rec)
[1] "id"     "age"    "vitals" "visits"
> str(rec)
List of 4
 $ id    : chr "P-104"
 $ age   : int 52
 $ vitals: Named num [1:3] 138 86 74
  ..- attr(*, "names")= chr [1:3] "sbp" "dbp" "pulse"
 $ visits:List of 2
  ..$ :List of 2
  .. ..$ reason: chr "check-up"
  .. ..$ fee   : num 300
  ..$ :List of 2
  .. ..$ reason: chr "fever"
  .. ..$ fee   : num 450

length(rec) is 4, the number of top-level elements, whatever each one holds, and names(rec) lists their names. The record is a nested listnested list: A list with other lists among its elements, such as a record whose visits element holds one list per visit. str() shows each level with one more ..$ of indentation., because visits is a list of two lists. Printing a nested list shows every value, level by level, which gets long quickly; str(), whose help page says it is especially well suited to compactly display nested lists, prints one line per element instead. Each line gives the type and the first values, and each extra ..$ is one more level of nesting. Run str() on any object you did not build yourself before you index into it.

The same tool reads what R hands back. A fitted model is a list:

from session import run, version

version()
# R reads model_list.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f model_list.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f model_list.R
> # A fitted model is a list too.
> fit <- lm(dist ~ speed, data = cars)
> is.list(fit)
[1] TRUE
> length(fit)
[1] 12
> str(fit, max.level = 1, give.attr = FALSE)
List of 12
 $ coefficients : Named num [1:2] -17.58 3.93
 $ residuals    : Named num [1:50] 3.85 11.85 -5.95 12.05 2.12 ...
 $ effects      : Named num [1:50] -303.914 145.552 -8.115 9.885 0.194 ...
 $ rank         : int 2
 $ fitted.values: Named num [1:50] -1.85 -1.85 9.95 9.95 13.88 ...
 $ assign       : int [1:2] 0 1
 $ qr           :List of 5
 $ df.residual  : int 48
 $ xlevels      : Named list()
 $ call         : language lm(formula = dist ~ speed, data = cars)
 $ terms        :Classes 'terms', 'formula'  language dist ~ speed
 $ model        :'data.frame':	50 obs. of  2 variables:
> fit$coefficients
(Intercept)       speed 
 -17.579095    3.932409 

lm() returned a list of 12 elements, among them the coefficients, the residuals and the QR decomposition, and fit$coefficients takes one of them out. str(fit, max.level = 1) stops after the top level, which keeps the output to one screen. Fitted models come back in the modelling module of this track, read the same way.

Single brackets keep a list, double brackets take the element out

rec["vitals"] and rec[["vitals"]] look alike and return different things. Single brackets select elements and keep them in a list, so rec["vitals"] is a list of length 1. Double brackets and $ take one element out of the list, so rec[["vitals"]] is the numeric vector itself. Step through what each form reaches:

The run of the same calls, then arithmetic on two of the results:

from session import run, version

version()
# R reads access.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f access.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f access.R
> rec <- list(
+   id = "P-104",
+   age = 52L,
+   vitals = c(sbp = 138, dbp = 86, pulse = 74),
+   visits = list(
+     list(reason = "check-up", fee = 300),
+     list(reason = "fever", fee = 450)
+   )
+ )
> rec["vitals"]
$vitals
  sbp   dbp pulse 
  138    86    74 

> class(rec["vitals"])
[1] "list"
> rec[["vitals"]]
  sbp   dbp pulse 
  138    86    74 
> class(rec[["vitals"]])
[1] "numeric"
> rec$vitals["sbp"]
sbp 
138 
> rec[["visits"]][[2]][["reason"]]
[1] "fever"
> rec[[c(4, 2, 1)]]
[1] "fever"
> # A name that is not there, and a position that is not there.
> rec[["weight"]]
NULL
> try(rec[[7]])
Error in rec[[7]] : subscript out of bounds
> # Arithmetic on what single brackets return.
> try(rec["vitals"] * 2)
Error in rec["vitals"] * 2 : non-numeric argument to binary operator
> rec[["vitals"]] * 2
  sbp   dbp pulse 
  276   172   148 

An Introduction to R calls the difference very important: [[ ]] selects a single element, while [ ] is the general subscripting operator, which returns a sub-list. So class(rec["vitals"]) is "list", and class(rec[["vitals"]]) is "numeric". Use [ ] to keep several elements as a list, as in rec[c("id", "age")], and [[ ]] or $ to work with the value of one.

Nested values take one step per level: rec[["visits"]][[2]][["reason"]] is the reason of the second visit. The help page of Extract adds that [[ ]] applies recursively when it is given a vector, so rec[[c(4, 2, 1)]] walks the same path by position. A missing element behaves differently by kind of index: rec[["weight"]] returns NULL, because the list has no element of that name, while rec[[7]] stops with “subscript out of bounds”.

The dollar sign matches partial names; double brackets do not

l <- list(alpha = 1, beta = 2) has no element called al, yet l$al returns 1. $ accepts the start of a name when only one name starts that way, a rule called partial matchingpartial matching: Accepting the start of a name as the whole name when exactly one name starts that way. $ does it on lists and data frames, so l$al can return l$alpha; [[ ]] does not unless exact = FALSE., and the help page of Extract states it exactly: x$name is equivalent to x[["name", exact = FALSE]]. The run tries both kinds of brackets, a data frame and a name that matches two elements:

from session import run, version

version()
# R reads partial.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f partial.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f partial.R
> l <- list(alpha = 1, beta = 2)
> l$al
[1] 1
> l[["al"]]
NULL
> l[["al", exact = FALSE]]
[1] 1
> d <- data.frame(alpha = 1:2)
> d$al
[1] 1 2
> # Two names start with "al", so nothing matches.
> two <- list(alpha = 1, alto = 2)
> two$al
NULL
> # Ask R to warn about every partial match.
> options(warnPartialMatchDollar = TRUE)
> l$al
[1] 1
Warning message:
In l$al : partial match of 'al' to 'alpha'
> d$al
[1] 1 2
Warning message:
In d$al : partial match of 'al' to 'alpha'

In R 4.6.1 nothing warns by default. options(warnPartialMatchDollar = TRUE), which its help page describes as a warning whenever $ uses partial matching, makes the last two lines of the run say “partial match of ‘al’ to ‘alpha’”. Turn it on while you develop a script, and write [[ ]] inside functions, where a name may come from a variable: a typo then gives NULL instead of the wrong element.

Assigning NULL deletes an element; list(NULL) stores one

The same brackets work on the left of <-, and one value behaves differently there: NULLNULL: R's object for nothing, of length 0. Assigning NULL to a list element deletes the element, while list(NULL) stores an element that holds NULL., R’s object for nothing. Assigning NULL to an element deletes the element, so the list gets shorter; storing NULL as a value takes a list that holds it. Step through four changes to a small list:

The run of those changes, and a merge of settings:

from session import run, version

version()
# R reads modify.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f modify.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f modify.R
> s <- list(a = 1, b = "x", c = TRUE)
> s$b <- NULL
> str(s)
List of 2
 $ a: num 1
 $ c: logi TRUE
> s["c"] <- list(NULL)
> str(s)
List of 2
 $ a: num 1
 $ c: NULL
> s$d <- 5
> names(s)
[1] "a" "c" "d"
> length(list(1, NULL, 3))
[1] 3
> # Settings: defaults, then what a user changed.
> defaults <- list(digits = 2, plot = list(width = 6, height = 4), title = "Report")
> user <- list(plot = list(width = 8))
> str(modifyList(defaults, user))
List of 3
 $ digits: num 2
 $ plot  :List of 2
  ..$ width : num 8
  ..$ height: num 4
 $ title : chr "Report"
> str(c(defaults, user))
List of 4
 $ digits: num 2
 $ plot  :List of 2
  ..$ width : num 6
  ..$ height: num 4
 $ title : chr "Report"
 $ plot  :List of 1
  ..$ width: num 8
> c(defaults, user)$plot$width
[1] 6

s$b <- NULL removed b. s["c"] <- list(NULL) kept c and made it hold NULL: single brackets on the left take a list of values, and the one value here is NULL. Assigning to a name that is not there, s$d <- 5, adds an element at the end. And list(1, NULL, 3) has length 3, because inside list() a NULL is an element like any other.

Settings are where this matters most. modifyList(defaults, user) merges a list of user choices into the defaults, level by level: its help page says it changes the matching elements at each level and adds new ones, so the user’s width = 8 replaced the default 6 while height = 4 stayed. c(defaults, user) only appends, and the result has two elements called plot; $ returns the first, so c(defaults, user)$plot$width is still the default 6, and the user’s choice is silently ignored.

unlist() flattens one record; a row builder flattens many

unlist() turns a nested list into one atomic vector and builds names from the path: unlist(list(a = 1, b = list(c = 2, d = 3))) gives a, b.c and b.d. The help page of unlist says the elements are coerced to a common type, which is why a record that mixes text and numbers flattens to text: "P-104", "138" and "86". That is the coercioncoercion: Converting values from one type to another. c() does it by itself when it combines values of different types; as.numeric(), as.character() and the other as.*() functions do it on request. More in “Atomic vectors, types and coercion” of c(), and it is fine for one record you want to look at, not for numbers you want to compute with.

For many records, the aim is a data frame with one row per record and one column per field. The shortcut, as.data.frame() of each record stacked with rbind(), fails as soon as the records disagree. Step through the route that works:

The run tries the shortcut, then builds the rows:

from session import run, version

version()
# R reads flatten.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f flatten.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f flatten.R
> unlist(list(a = 1, b = list(c = 2, d = 3)))
  a b.c b.d 
  1   2   3 
> unlist(list(id = "P-104", vitals = c(sbp = 138, dbp = 86)))
        id vitals.sbp vitals.dbp 
   "P-104"      "138"       "86" 
> # Three visit records; two of them miss a field.
> recs <- list(
+   list(id = "P-101", age = 34, sbp = 122),
+   list(id = "P-102", age = 61),
+   list(id = "P-103", sbp = 141, age = 47)
+ )
> try(do.call(rbind, lapply(recs, as.data.frame)))
Error in rbind(deparse.level, ...) : 
  numbers of columns of arguments do not match
> fields <- c("id", "age", "sbp")
> row_of <- function(r) {
+   vals <- lapply(fields, function(f) if (is.null(r[[f]])) NA else r[[f]])
+   names(vals) <- fields
+   as.data.frame(vals)
+ }
> visits <- do.call(rbind, lapply(recs, row_of))
> visits
     id age sbp
1 P-101  34 122
2 P-102  61  NA
3 P-103  47 141
> str(visits)
'data.frame':	3 obs. of  3 variables:
 $ id : chr  "P-101" "P-102" "P-103"
 $ age: num  34 61 47
 $ sbp: num  122 NA 141

row_of() asks every record for the same fields, in the same order, and uses NA where r[[f]] is NULL, so every row has the same three columns and a missing reading becomes a missing value instead of a missing column. lapply() calls it once per record, and do.call(rbind, rows), which constructs a call from a function and a list of arguments, as its help page puts it, hands the three one-row data frames to rbind() as separate arguments. str() confirms the types: id is text, and age and sbp are numbers, with one NA.

Practise: predict, then check

Eight lines poke at one small nested list. Note what you expect from each of them, and only then look.

from session import run, version

version()
# R reads practice.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f practice.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f practice.R
> x <- list(a = 1:3, b = list(c = "z"))
> length(x)
[1] 2
> class(x["a"])
[1] "list"
> x$b$c
[1] "z"
> x$a[2]
[1] 2
> x$b <- NULL
> length(x)
[1] 1
> unlist(list(p = 1, q = list(r = "2")))
  p q.r 
"1" "2" 

The list has two top-level elements, however long they are; single brackets returned a list; $ went down two levels to "z"; x$a[2] indexed the vector inside; assigning NULL deleted b; and unlist() coerced 1 to text to sit beside "2", with the names p and q.r.

Now the questions, beginning with the idea you have answered least well, each answer explained at once:

Interview questions

Warm-up: why can l$al return a value when the list has no element called al, and how do you avoid it?

$ uses partial matching: when the name you type is the start of exactly one element’s name, it returns that element, so with l <- list(alpha = 1), l$al gives 1. The help page of Extract says x$name is equivalent to x[["name", exact = FALSE]]. Data frames behave the same way, so d$al can return the column alpha. If two names start that way, $ returns NULL. Nothing warns by default, so a typo or a shortened name can read the wrong element without a sign. To avoid it, use [[ ]] in code, which matches whole names only and returns NULL for a name that is not there, and turn on options(warnPartialMatchDollar = TRUE) while developing, which makes every partial match print a warning.

Key takeaways

• A list holds objects of any type and length, other lists included, and str() shows a nested list’s shape one line per element.
• [ ] returns a smaller list, while [[ ]] and $ return one element itself; [[ ]] with a vector walks a nested path, and a missing name gives NULL.
• $ matches partial names silently, on lists and data frames; [[ ]] matches whole names only, and options(warnPartialMatchDollar = TRUE) warns about every partial match.
• Assigning NULL deletes an element, [ ] <- list(NULL) stores NULL in one, and modifyList() merges nested settings, where c() would only append duplicates.
• unlist() flattens one record into a named vector of one type; to flatten many, build one row per record with the same fields and stack the rows with do.call(rbind, rows).