Scripts, projects and file paths that travel

A script that starts with setwd("C:/Users/asha/Desktop/survey") runs on one computer only. On every other one it stops on its first line with “cannot change working directory”, before it has read a single row. The fix is a habit rather than a tool: one folder per analysis, every path relative to that folder, and scripts that run from their first line to their last on their own.

Below you lay out a project folder, run scripts with source() and with Rscript and give them arguments, build paths that work on Windows, macOS and Linux, list files without being caught out by a regular expression, and read settings from the environment instead of writing them into the code.

A project folder keeps raw data, scripts and output apart

Put everything one analysis needs in one folder, and give each kind of file its own place inside it. Follow the data of a small clinic-visits analysis through such a folder:

The rule behind the layout is that data only ever changes by running code. Raw data stays as it arrived, cleaned data and output are made by scripts, and so anyone can delete them and make them again. This script creates the folders, with a raw file and a README, and lists what it made:

# One folder per analysis: raw data, cleaned data, scripts and output apart.
project <- "clinic-visits"
for (part in c("data/raw", "data/clean", "R", "output")) {
  dir.create(file.path(project, part), recursive = TRUE)
}
writeLines(
  c("site,visits", "north,12", "south,7"),
  file.path(project, "data", "raw", "visits.csv")
)
writeLines("Where the data came from, and when.", file.path(project, "README.txt"))

# Every folder and file of the project, relative to its top folder.
list.files(project, recursive = TRUE, include.dirs = TRUE)

from session import run, version

version()
run("""
Rscript project_skeleton.R
""")

# R version 4.6.1 -- "Happy Hop"
$ Rscript project_skeleton.R
[1] "data"                "data/clean"          "data/raw"           
[4] "data/raw/visits.csv" "output"              "R"                  
[7] "README.txt"         

list.files(project, recursive = TRUE, include.dirs = TRUE) lists every folder and file under clinic-visits, as paths relative to it: data/raw/visits.csv is a path from the project’s top folder, which is the point of the layout.

Scripts find these files from the working directoryworking directory: The folder that relative paths start from. getwd() shows it; opening a project in RStudio, or starting R or Rscript in a folder, sets it., the folder that relative paths start from. Opening a project in RStudio sets the working directory to the project’s folder and starts a new R session; from a terminal, start R or Rscript in the project’s top folder. Two more habits keep an analysis honest:

• Restart R often, and do not save the workspace. At the end of a session R offers to save every object into a file called .RData, and the next R started in that folder loads it again by itself. Objects that come back that way, rather than from your scripts, make results that no script can reproduce. Answer no, or start R with --no-save.
• Name files plainly. The tidyverse style guide asks for .R file names in lower case, with words joined by - or _ and no spaces or symbols, so that they work in every system’s paths.

source() runs a script in your session; Rscript runs it on its own

There are two ways to run a script, and they differ in where the results end up. source("file.R") runs every line of the file in your current session, and what the script makes stays in your workspace. RscriptRscript: The program that runs an R script from a terminal in a new R session, which ends with the script. Words after the file name reach commandArgs(). runs the file in a new R session, which ends when the script ends. This script reads the visits and keeps their total:

# Reads the raw visits and keeps their total in total_visits.
visits <- read.csv(file.path("data", "raw", "visits.csv"))
total_visits <- sum(visits$visits)

The session below sources it at the console, then runs the same file with Rscript, then asks a new R whether the total exists:

from session import run, version

version()
# source() inside a console session, then the same script run on its own with Rscript, then a new R that looks for
# what the script made.
run("""
R --vanilla --quiet -f source_demo.R
Rscript R/count_sites.R
Rscript -e 'exists("total_visits")'
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f source_demo.R
> # source() runs every line of a script in this session.
> source(file.path("R", "count_sites.R"))
> # What the script made stays in the workspace.
> total_visits
[1] 43
> ls()
[1] "total_visits" "visits"      
$ Rscript R/count_sites.R
$ Rscript -e 'exists("total_visits")'
[1] FALSE

After source() the workspace holds total_visits, 43, and visits. Rscript R/count_sites.R prints nothing, because the script has only assignments, and leaves nothing behind: the new R answers FALSE. Use source() while you work on an analysis, and Rscript for runs that must not depend on anything in your session, such as the final run of a report.

A script run by Rscript can take command-line argumentscommand-line argument: A word typed after a script's name when it is run, such as 10 in Rscript summarise.R data.csv 10. R receives every argument as text., words after its file name, which commandArgs(trailingOnly = TRUE) returns. Step through a run of summarise.R, which keeps the sites with at least a given number of visits:

# Usage: Rscript summarise.R <csv file> <minimum visits>
args <- commandArgs(trailingOnly = TRUE)
str(args)
if (length(args) != 2) {
  stop("give a CSV file and a minimum number of visits")
}
path <- args[1]
minimum <- suppressWarnings(as.numeric(args[2]))
if (is.na(minimum)) stop("the minimum must be a number, not ", sQuote(args[2]))
if (!file.exists(path)) stop("there is no file ", path)

visits <- read.csv(path)
busy <- visits[visits$visits >= minimum, ]
dir.create("output", showWarnings = FALSE)
out <- file.path("output", paste0("busy-sites-", minimum, ".csv"))
write.csv(busy, out, row.names = FALSE)
cat("Read", nrow(visits), "sites; wrote", nrow(busy), "with at least", minimum, "visits to", out, "\n")

from session import run, version

version()
# Two good runs, the file the first one wrote, then two runs that the script stops on purpose.
run("""
Rscript summarise.R data/raw/visits.csv 10
cat output/busy-sites-10.csv
Rscript summarise.R data/raw/visits.csv 8
!Rscript summarise.R data/raw/visits.csv ten
!Rscript summarise.R
""")

# R version 4.6.1 -- "Happy Hop"
$ Rscript summarise.R data/raw/visits.csv 10
 chr [1:2] "data/raw/visits.csv" "10"
Read 4 sites; wrote 2 with at least 10 visits to output/busy-sites-10.csv 
$ cat output/busy-sites-10.csv
"site","visits"
"north",12
"east",15
$ Rscript summarise.R data/raw/visits.csv 8
 chr [1:2] "data/raw/visits.csv" "8"
Read 4 sites; wrote 3 with at least 8 visits to output/busy-sites-8.csv 
$ Rscript summarise.R data/raw/visits.csv ten
 chr [1:2] "data/raw/visits.csv" "ten"
Error: the minimum must be a number, not ‘ten’
Execution halted
[exit status 1]
$ Rscript summarise.R
 chr(0) 
Error: give a CSV file and a minimum number of visits
Execution halted
[exit status 1]

The first run kept the 2 sites with at least 10 visits, north and east, and the second the 3 with at least 8. With ten, and with no arguments at all, the script stopped on purpose with a message that says what is wrong, and Rscript exited with status 1, which tells a scheduler or another script that the run failed.

Paths that work on every computer

A path written as one piece of text, "data\\raw\\visits.csv" or "/Users/asha/survey/data.csv", carries the habits of one system or the folders of one person. Build paths from their parts instead, relative to the project’s top folder:

from session import run, version

version()
run("""
R --vanilla --quiet -f paths.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f paths.R
> # file.path() joins the parts with "/", which R also accepts on Windows.
> raw <- file.path("data", "raw", "visits.csv")
> raw
[1] "data/raw/visits.csv"
> # Take a path apart without cutting text by hand.
> basename(raw)
[1] "visits.csv"
> dirname(raw)
[1] "data/raw"
> tools::file_ext(raw)
[1] "csv"
> tools::file_path_sans_ext(basename(raw))
[1] "visits"
> # A relative path starts from the working directory.
> file.exists(raw)
[1] TRUE
> file.exists(file.path("data", "raw", "missing.csv"))
[1] FALSE

file.path() joins the parts with /, and its help page notes that /, not \, is the separator it uses on Windows too, so the same line works everywhere. The other functions take a path apart without any cutting of text:

A relative pathrelative path: A path that starts from the working directory, such as data/raw/visits.csv, rather than from the top of the disk. such as data/raw/visits.csv starts from the working directory, so it works on any computer as long as R starts in the project’s top folder. The here package, on CRAN, goes one step further: it builds paths from a project’s top folder wherever a script runs, and its documentation calls setwd() “fragile and dependent on the way you organize your files”.

list.files() matches a regular expression unless you ask for fixed

list.files() lists the files of a folder, sorted alphabetically, and its pattern argument picks some of them. pattern is a regular expressionregular expression: A pattern language for text, in which . matches any character, ^ the start and $ the end. list.files() reads its pattern argument as one., not plain text, and that catches people out. Here are five file names and four ways to ask for the CSV files:

from session import run, version

version()
run("""
R --vanilla --quiet -f list_files.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f list_files.R
> # Five files in a folder called inbox.
> files <- c("visits.csv", "visits.csv.bak", "visits_csv.txt", "old-visits.csv", "notes.txt")
> dir.create("inbox")
> invisible(file.create(file.path("inbox", files)))
> # pattern is a regular expression, in which "." matches any character.
> list.files("inbox", pattern = "visits.csv")
[1] "old-visits.csv" "visits_csv.txt" "visits.csv"     "visits.csv.bak"
> # Escape the dot and anchor the end: only names that end in .csv.
> list.files("inbox", pattern = "\\.csv$")
[1] "old-visits.csv" "visits.csv"    
> # fixed = TRUE, new in R 4.6.0, matches the text exactly as written.
> list.files("inbox", pattern = "visits.csv", fixed = TRUE)
[1] "old-visits.csv" "visits.csv"     "visits.csv.bak"
> # full.names = TRUE gives paths you can open.
> list.files("inbox", pattern = "\\.csv$", full.names = TRUE)
[1] "inbox/old-visits.csv" "inbox/visits.csv"    

In a regular expression . matches any character, and a pattern may match anywhere in a name. So pattern = "visits.csv" also matched visits_csv.txt, through its underscore, and visits.csv.bak, which merely contains the text. Two fixes, for two needs:

• A pattern that means what it says: "\\.csv$", with the dot escaped as \\. and $ for the end of the name, lists only the names that end in .csv.
• Plain text: fixed = TRUE, an argument new in R 4.6.0, matches the pattern exactly as written. It no longer matched visits_csv.txt, but it still matched visits.csv.bak, which contains the text.

full.names = TRUE adds the folder to each name, giving paths you can open, and recursive = TRUE looks inside subfolders, as the project skeleton did.

Settings and other programs: Sys.getenv() and system2()

A script should not change from one computer to the next, but some settings do: a folder outside the project, a threshold that differs between runs, a password. Read them from environment variablesenvironment variable: A named setting that the operating system passes to the programs it starts. R reads one with Sys.getenv() and always gets text., which the system or the person running the script sets, and run other programs with system2():

from session import run, version

version()
run("""
R --vanilla --quiet -f settings.R
""")

# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f settings.R
> # Sys.getenv() reads an environment variable; unset is the value when it is missing.
> Sys.getenv("CLINIC_MIN_VISITS", unset = "10")
[1] "10"
> # Set it for this session, then read it again.
> Sys.setenv(CLINIC_MIN_VISITS = "8")
> as.numeric(Sys.getenv("CLINIC_MIN_VISITS"))
[1] 8
> # system2() runs another program; stdout = TRUE returns what it printed.
> system2("Rscript", c("-e", shQuote("cat(6 * 7)")), stdout = TRUE)
[1] "42"

Sys.getenv("CLINIC_MIN_VISITS", unset = "10") returned "10" because the variable was not set; after Sys.setenv() it returned "8", text again, which as.numeric() turns into a number. system2() ran a second R, and with stdout = TRUE it returned what that program printed as text, "42".

Sys.setenv() lasts for the current R session and the programs it starts, and changes nothing outside them. To set a variable for every session, write a line such as CLINIC_MIN_VISITS=8 in a file called .Renviron, which R reads when it starts, from the current folder or else from your home folder. R --vanilla skips that file, and a .Renviron that holds a secret stays out of version control.

Practise: a project that runs anywhere

Build a small project on your own computer, in order:

1. Make a folder called clinic-visits with data/raw, data/clean, R and output inside it, and put a short CSV file of sites and visits into data/raw.
2. Write R/summarise.R like the script above, with file.path() for every path and no setwd().
3. In a terminal in the project’s top folder, run Rscript R/summarise.R data/raw/visits.csv 10, then with ten and check that it stops with your message.
4. Copy the folder somewhere else, run the same command there, and compare the files in output.

Finally, the self-check covers projects, scripts, paths, file patterns and settings. It picks the idea you are least sure of, explains every answer as soon as you give it, and ends with a summary that links each idea to the part to reread:

Interview questions

Warm-up: why should an analysis script not call setwd() with an absolute path?

Because the path exists only on the computer where the script was written. setwd("C:/Users/asha/Desktop/survey") stops with “cannot change working directory” on every other computer, before any work is done, and on the same computer it breaks as soon as the folder moves or is renamed. It also hides an assumption: every relative path after it depends on that one line. The portable way is to start R in the project’s top folder, which opening an RStudio project does and a terminal does when you run Rscript there, and to build every path from the project’s top with file.path(). The here package goes one step further and finds the project’s top folder wherever the script runs. setwd() is still fine at the console for a quick look, as long as no script depends on it.

Key takeaways

• One folder per analysis keeps raw data untouched in data/raw, scripts in R/ and everything they make in data/clean and output, so a copy of the folder can rerun the whole analysis.
• source() runs a script in your session and leaves its objects in your workspace; Rscript runs it in a new session that keeps nothing, and exits with status 1 when the script stops with an error.
• commandArgs(trailingOnly = TRUE) returns a script’s arguments as text, which must be converted and checked before they are used.
• file.path() and relative paths from the project’s top folder work on every system, and basename(), dirname() and tools::file_ext() take paths apart, where setwd() with an absolute path breaks on the next computer.
• list.files(pattern = ) reads a regular expression, in which . matches any character; anchor and escape it, or use fixed = TRUE from R 4.6.0.
• Sys.getenv() reads settings and secrets from the environment, and system2() runs another program and can return what it printed.