Missing values, NaN, Inf and NULL

mean(c(72, NA, 85)) is NA: one missing mark makes the whole average missing, and R is right not to guess. Missing values cause most of the quiet mistakes in data work: a test with == that never finds them, a count that leaves them out without saying so, and a filter that adds rows of NA to a table.

On runs of R 4.6.1 you will tell NA, NaN, Inf and NULL apart and pick the right test for each, handle missing values in summaries with na.rm, complete.cases() and na.omit(), filter a table without rows of NA, and report missing values so that a reader can see them.

NA spreads through a calculation unless the answer cannot depend on it

NA means “not available”: a value that exists but is not known. The R Language Definition says numeric and logical calculations with NA generally return NA, and that is what keeps an unknown from turning into a confident answer. mean(c(72, NA, 85)) is NA, because the mean depends on the missing mark; na.rm = TRUE asks R to leave missing values out, and gives 78.5 from the other two.

from session import run, version

version()
# R reads na_basics.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f na_basics.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f na_basics.R
> # One missing mark makes the average missing.
> mean(c(72, NA, 85))
[1] NA
> mean(c(72, NA, 85), na.rm = TRUE)
[1] 78.5
> NA + 1
[1] NA
> NA * 0
[1] NA
> # Unless the answer is the same whatever NA stands for.
> NA^0
[1] 1
> NA & FALSE
[1] FALSE
> # NA has a type, like every value.
> typeof(NA)
[1] "logical"
> typeof(NA_real_)
[1] "double"
> typeof(NA_character_)
[1] "character"
> # x == NA is NA everywhere; is.na() is the test.
> x <- c(4, NA, 7, NA)
> x == NA
[1] NA NA NA NA
> is.na(x)
[1] FALSE  TRUE FALSE  TRUE
> sum(is.na(x))
[1] 2
> anyNA(x)
[1] TRUE
> # Text is different: paste() writes NA as two letters.
> paste("ward", NA)
[1] "ward NA"
> nchar(c("ab", NA))
[1]  2 NA

The run shows the exceptions and the traps:

• An answer that cannot depend on the unknown is given. NA^0 is 1, because the help page ?Arithmetic says y ^ 0 is 1 always, and NA & FALSE is FALSE. NA * 0 stays NA, since the missing value could be Inf, and Inf * 0 has no answer.
• NA has a type. A bare NA is logical, and NA_real_, NA_integer_ and NA_character_ are missing values of the other types, which is why one gap never changes a vector’s type.
• x == NA is never the test. Comparing anything with an unknown gives an unknown, so x == NA is NA for every element. is.na(x) gives TRUE or FALSE for each, sum(is.na(x)) counts the gaps and anyNA(x) asks whether there is at least one.
• Text turns NA into letters. paste("ward", NA) is "ward NA", a string that is no longer missing, while nchar() keeps a missing string missing. Check is.na() before you build labels from data.

NaN, Inf and NULL are not NA, and each has its own test

0 / 0 is NaN, “not a number”, and 1 / 0 is Inf. Both are values of the IEEE 754 arithmetic that R’s doubles follow: NaN is the result of arithmetic with no defined answer, such as 0 / 0, sqrt(-1) or Inf - Inf, while Inf and -Inf stand for results too large for a double, such as 10^400, and for a non-zero number divided by zero, such as 1 / 0. NULL is something else again, the absence of any value: it has length 0, and inside c() it disappears. R prints four tests against the five kinds of value:

from session import run, version

version()
# R reads special_values.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f special_values.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f special_values.R
> # NaN, "not a number", comes from arithmetic with no answer.
> 0 / 0
[1] NaN
> sqrt(-1)
[1] NaN
Warning message:
In sqrt(-1) : NaNs produced
> Inf - Inf
[1] NaN
> Inf * 0
[1] NaN
> # Inf stands for results beyond the largest double.
> 1 / 0
[1] Inf
> -1 / 0
[1] -Inf
> 10^400
[1] Inf
> # Four tests against five kinds of value.
> v <- c(1, NA, NaN, Inf, -Inf)
> names(v) <- c("1", "NA", "NaN", "Inf", "-Inf")
> rbind(is.na = is.na(v), is.nan = is.nan(v),
+       is.finite = is.finite(v), is.infinite = is.infinite(v))
                1    NA   NaN   Inf  -Inf
is.na       FALSE  TRUE  TRUE FALSE FALSE
is.nan      FALSE FALSE  TRUE FALSE FALSE
is.finite    TRUE FALSE FALSE FALSE FALSE
is.infinite FALSE FALSE FALSE  TRUE  TRUE
> # The largest value of nothing is -Inf, with a warning.
> max(numeric(0))
[1] -Inf
Warning message:
In max(numeric(0)) : no non-missing arguments to max; returning -Inf
> # NULL is no value at all: length 0, gone inside c().
> length(NULL)
[1] 0
> c(1, NULL, 2)
[1] 1 2
> is.null(NULL)
[1] TRUE
> is.na(NULL)
logical(0)

Read the table by column. is.na() is TRUE for NA and for NaN, which R treats as a kind of missing value, while is.nan() is TRUE for NaN only, so is.nan(NA) is FALSE. is.finite() is TRUE for ordinary numbers alone. max(numeric(0)) returns -Inf with a warning, the maximum of nothing, so a group with no values can put an infinity into a table without an error. NULL passes none of these tests: is.na(NULL) is logical(0), an empty answer, and is.null() is the test for it.

Which test you need depends on which values you want to find. Say which, and the table rules out the rest:

Summaries leave missing values out only when you ask

sum(), mean() and max() return NA when their input has a gap, until you add na.rm = TRUE. That default is the safe one: leaving values out changes the answer, so R makes you ask for it. For a whole table, a row with no missing value in any column is a complete casecomplete case: A row of a table with no missing value in any column. complete.cases() marks such rows, and na.omit() keeps only them, so a complete-case analysis uses them alone.: complete.cases() marks them, and na.omit() keeps only them and records the rows it dropped in an attribute called na.action.

A data frame, which a later lesson covers in full, is a table whose columns are vectors of one length, and survey$income picks one column. Here is a missingness report for a seeded survey of 200 people:

from session import run, version

version()
# R reads missingness_report.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f missingness_report.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f missingness_report.R
> # A seeded survey of 200 people, with gaps in every column.
> set.seed(7)
> survey <- data.frame(
+   age = sample(c(18:70, NA), 200, replace = TRUE),
+   income = ifelse(runif(200) < 0.12, NA,
+                   round(rlnorm(200, 10, 0.5))),
+   district = sample(c("North", "South", NA), 200,
+                     replace = TRUE, prob = c(0.5, 0.45, 0.05))
+ )
> # Missing values per column, as a count and a percentage.
> n_missing <- colSums(is.na(survey))
> n_missing
     age   income district 
       3       23       10 
> round(100 * n_missing / nrow(survey), 1)
     age   income district 
     1.5     11.5      5.0 
> # The rows a complete-case analysis would keep.
> sum(complete.cases(survey))
[1] 166
> nrow(na.omit(survey))
[1] 166
> # table() leaves NA out unless you ask for it.
> table(survey$district)

North South 
  108    82 
> table(survey$district, useNA = "ifany")

North South  <NA> 
  108    82    10 
> # summary() in R 4.6 counts missing values as NAs,
> # and describes text in more detail.
> summary(survey$income)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.     NAs 
   6318   15493   21535   25232   30896  101529      23 
> summary(survey$district)
   Length  N.unique   N.blank Min.nchar Max.nchar       NAs 
      200         2         0         5         5        10 

No column misses more than 11.5 % of its values, yet a complete-case analysis keeps only 166 of the 200 rows and drops 34, 17 %, because the gaps fall on different rows. Report that number before you analyse what na.omit() left. table() leaves NA out unless you ask with useNA = "ifany", which shows the 10 people with no district.

A filter with NA in its condition adds rows of NA

scores[scores$score > 15, ] should return the students who scored above 15. When some scores are missing, the condition is NA for them, and the row index of a data frame treats NA the way x[NA] did for a vector in the indexing lesson: it selects a row whose contents are unknown, and R fills it with NA. Step through the filter:

The same filter, the rows it counts, and the two forms that avoid the rows of NA, typed at the console:

from session import run, version

version()
# R reads na_rows.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f na_rows.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f na_rows.R
> scores <- data.frame(id = 1:5, score = c(12, NA, 18, 20, NA))
> # The condition is NA where the score is missing ...
> scores$score > 15
[1] FALSE    NA  TRUE  TRUE    NA
> # ... and each NA in a row index gives a row of NA.
> scores[scores$score > 15, ]
     id score
NA   NA    NA
3     3    18
4     4    20
NA.1 NA    NA
> # nrow() counts the made-up rows as well.
> nrow(scores[scores$score > 15, ])
[1] 4
> # subset() and which() keep only the rows that are TRUE.
> subset(scores, score > 15)
  id score
3  3    18
4  4    20
> scores[which(scores$score > 15), ]
  id score
3  3    18
4  4    20

The result has four rows, two of them, named NA and NA.1, made of nothing. nrow() counts them, 4 rows for 2 real students, and a mean of the column then includes the gaps again. subset() keeps only the rows where its condition is TRUE, which its help page states, and which() turns the condition into the positions of its TRUE values, so both give the two real rows. Use one of them whenever the column you filter on can be missing.

Practise: predict, then check

One vector with a missing value and a zero touches every idea above. Note your answers, then run the lines.

from session import run, version

version()
# R reads practice.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f practice.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f practice.R
> x <- c(3, NA, 0, 6)
> sum(x)
[1] NA
> sum(x, na.rm = TRUE)
[1] 9
> x / 0
[1] Inf  NA NaN Inf
> is.na(x / 0)
[1] FALSE  TRUE  TRUE FALSE
> mean(is.na(x))
[1] 0.25
> length(c(x, NULL))
[1] 4

The gap made the plain sum NA; na.rm = TRUE added the other three; dividing by zero gave Inf twice, NA for the gap and NaN for 0 / 0; is.na() is TRUE for both the NA and the NaN; a quarter of the values are missing; and the NULL added nothing.

For the questions, the first comes from the idea you know least so far, every answer is explained on the spot, and the level rises with each right answer:

Interview questions

Warm-up: what is the difference between NA and NULL in R?

NA is a value that is missing: it holds a place in a vector, has a type (NA is logical, and NA_real_, NA_integer_ and NA_character_ are its numeric, integer and text forms), and counts towards the length, so length(c(1, NA)) is 2. Calculations with it usually give NA, and is.na() finds it. NULL is no value at all: it has length 0, it has no type of the six atomic kinds, and inside c() it disappears, so c(1, NULL) is just 1. is.null() finds it, while is.na(NULL) returns an empty answer. In practice, NA marks a gap in data, such as a survey answer that was not given, while NULL marks the absence of an object, such as an optional argument that was not supplied.

Key takeaways

• NA is an unknown value with a type; most calculations with it return NA, except when the answer cannot depend on it, as with NA^0 and NA & FALSE, and is.na() is the only reliable test.
• NaN comes from arithmetic with no answer and counts as missing for is.na(), Inf from overflow or division by zero, and NULL is no value at all, of length 0; max() of nothing is -Inf with a warning.
• Summaries return NA until you add na.rm = TRUE; complete.cases() and na.omit() work on whole rows, and a complete-case analysis can drop many more rows than any one column misses.
• Report missing values: sum(is.na(x)) and its share, table(x, useNA = "ifany"), and the NAs count of summary(), which R 4.6.0 relabelled from NA's.
• Filtering a data frame with a condition that is NA adds rows of NA; subset() and which() keep only the rows where the condition is TRUE.