R (4.6) Module 7 – Importing and exporting data
JSON, web APIs and responsible scraping
Turn JSON into R data frames with jsonlite, page through a web API with httr2, keep keys out of scripts, and scrape tables with rvest only where allowed.
What you will learn
- Convert between JSON and R lists or data frames with jsonlite, and say what each JSON shape becomes
- Request data from a web API with httr2, following its pages and its limits, from recorded responses
- Store API keys in environment variables and check that no script holds one
- Extract an HTML table with rvest only after the API, the site's terms and robots.txt allow it
Before you start
What you’ll learn · Before you start
What you will learn
- Convert between JSON and R lists or data frames with jsonlite, and say what each JSON shape becomes
- Request data from a web API with httr2, following its pages and its limits, from recorded responses
- Store API keys in environment variables and check that no script holds one
- Extract an HTML table with rvest only after the API, the site's terms and robots.txt allow it
Before you start
On this page
The World Bank’s indicators API answers a question about internet use in four South Asian countries in 3 pages of JSON, 20 records in all, with three values missing. A script that reads only the first page gets 8 records and no error; one that sends its key in a header it prints puts the key in every log.
On runs of R 4.6.1 with jsonlite 2.0.0, httr2 1.3.0 and rvest 1.0.5, you will turn JSON into vectors, lists and data frames and back, page through that recorded answer with httr2 as a script would against the live API, keep a key in an environment variable and check that no script holds one, and read a table from a web page only after the API, the site’s terms and its robots.txt have had their say.
JSON arrays and objects become vectors, lists and data frames
JSON: JavaScript Object Notation, a text format for data built from objects in braces, arrays in brackets, strings, numbers, true, false and null (RFC 8259). Most web APIs answer in it. has two containers: objects in braces, whose values have names, and arrays in brackets,
whose values are in order. RFC 8259 defines both, with strings, numbers, true, false and null as the values
inside them. jsonlite’s fromJSON() maps each shape onto the R structure that fits it best, the mapping its author
describes in the paper behind the package:
from session import run, version
version()
# R reads json_shapes.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f json_shapes.R
""")Output
# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f json_shapes.R
> library(jsonlite)
> # A JSON array of numbers, with a null:
> fromJSON('[61.5, 70, null]')
[1] 61.5 70.0 NA
> # A JSON object:
> str(fromJSON('{"id": "P-104", "age": 52, "tags": ["new", "urgent"]}'))
List of 3
$ id : chr "P-104"
$ age : int 52
$ tags: chr [1:2] "new" "urgent"
> # An array of objects, the shape most APIs answer in:
> visits <- fromJSON('[{"id": "P-101", "ward": "ICU", "fee": 1250},
+ {"id": "P-102", "ward": "General", "fee": null},
+ {"id": "P-103", "fee": 450}]')
> visits
id ward fee
1 P-101 ICU 1250
2 P-102 General NA
3 P-103 <NA> 450
> # The same text without simplifying: a list of lists.
> str(fromJSON('[{"id": "P-102", "fee": null}]', simplifyVector = FALSE))
List of 1
$ :List of 2
..$ id : chr "P-102"
..$ fee: NULL
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 json_shapes_session.py
An array of numbers became a numeric vector, with null as NA. An object became a named list. An array of objects
that share their names, the shape most APIs answer in, became a data frame with one row per object, and a name missing
from one object became NA there, as for the ward of P-103. simplifyVector = FALSE turns the mapping off and returns
the raw list: R's container for objects of any type and length, each element with an optional name. [ ] returns a smaller list, while [[ ]] and $ return one element itself. More in “Lists and nested data” of lists, with null as NULL. Records with something nested inside them go one
step further:
from session import run, version
version()
# R reads json_nested.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f json_nested.R
""")Output
# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f json_nested.R
> library(jsonlite)
> recs <- '[{"id": "P-101", "ward": {"name": "ICU", "floor": 2}, "codes": ["J10", "R50"]},
+ {"id": "P-102", "ward": {"name": "General", "floor": 1}, "codes": []}]'
> v <- fromJSON(recs)
> str(v)
'data.frame': 2 obs. of 3 variables:
$ id : chr "P-101" "P-102"
$ ward :'data.frame': 2 obs. of 2 variables:
..$ name : chr "ICU" "General"
..$ floor: int 2 1
$ codes:List of 2
..$ : chr "J10" "R50"
..$ : chr
> names(fromJSON(recs, flatten = TRUE))
[1] "id" "codes" "ward.name" "ward.floor"
> v$codes
[[1]]
[1] "J10" "R50"
[[2]]
character(0)
> lengths(v$codes)
[1] 2 0
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 json_nested_session.py
The object ward became a data frame inside the data frame, which flatten = TRUE turns into the columns
ward.name and ward.floor. The array codes cannot fit one cell, so it became a list column, one character vector
per record; lengths() counts 2 codes for P-101 and 0 for P-102. Step through the five shapes:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- An array of values. An array whose values are all numbers becomes a numeric vector, and the JSON null becomes NA, so [61.5, 70, null] is 61.5 70.0 NA.
- An object. An object becomes a named list, one element per name. The array of tags inside it becomes a character vector of two.
- An array of records. An array of objects that share names becomes a data frame with one row per object. A missing name or a null becomes NA, as for the fee of P-102 and the ward of P-103.
- An object inside a record. When every record holds an object, such as ward, the column is itself a data frame. flatten = TRUE turns it into the columns ward.name and ward.floor.
- An array inside a record. An array inside each record, such as codes, cannot fit one cell, so it becomes a list column. lengths() gives 2 codes for P-101 and 0 for P-102.
- Turning it off. simplifyVector = FALSE keeps every array as a list and every null as NULL, the raw shape that httr2's resp_body_json() also returns.
One string in an array of numbers
fromJSON('[1, "2", 3]') returns the text "1" "2" "3": a vector has one type, and the quoted "2" makes it
character, as c() would. Check the types after every read, with str(), before you compute with a column.
Checkpoint
Check: JSON shapes in R
toJSON() writes records, drops NA and rounds to four digits
Sending data to an API means writing JSON, and toJSON() makes four choices for you that an API may not share. The
run writes two fees, one of them NA, with a share of 1/3 and 2/3:
from session import run, version
version()
# R reads to_json.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f to_json.R
""")Output
# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f to_json.R
> library(jsonlite)
> fees <- data.frame(id = c("P-101", "P-102"), fee = c(1250, NA),
+ share = c(1/3, 2/3))
> toJSON(fees)
[{"id":"P-101","fee":1250,"share":0.3333},{"id":"P-102","share":0.6667}]
> toJSON(fees, na = "null")
[{"id":"P-101","fee":1250,"share":0.3333},{"id":"P-102","fee":null,"share":0.6667}]
> toJSON(fees, dataframe = "columns")
{"id":["P-101","P-102"],"fee":[1250,"NA"],"share":[0.3333,0.6667]}
> toJSON(fees, digits = NA, na = "null")
[{"id":"P-101","fee":1250,"share":0.333333333333333},{"id":"P-102","fee":null,"share":0.666666666666667}]
> toJSON(list(id = "P-101", fee = 1250))
{"id":["P-101"],"fee":[1250]}
> toJSON(list(id = "P-101", fee = 1250), auto_unbox = TRUE, pretty = TRUE)
{
"id": "P-101",
"fee": 1250
}
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 to_json_session.py
Default of toJSON() |
What it wrote | Change it with |
|---|---|---|
| A data frame becomes an array of records | [{"id":"P-101",…},{"id":"P-102",…}] |
dataframe = "columns" for one array per column |
An NA field is left out of its record |
P-102 has no fee at all |
na = "null" |
| Numbers keep at most 4 decimal places | 0.3333 for 1/3 |
digits = NA for full precision |
| Every vector is an array, even of length 1 | {"id":["P-101"]} |
auto_unbox = TRUE, or unbox() for one value |
Two of the rows are traps that leave no trace. The share 1/3 went out as 0.3333, so whatever reads it gets a
different number, and with dataframe = "columns" the missing fee went out as the string "NA", text inside a column
of numbers. pretty = TRUE only changes the layout, for people reading the JSON.
Key idea
Read what the receiving program expects, then say it in the call: toJSON(x, digits = NA, na = "null"), and
auto_unbox = TRUE when it wants single values rather than arrays of one.
Checkpoint
Check: Writing JSON
An API answers one page at a time
A web API: A service that answers requests sent to web addresses with data, usually JSON, rather than pages for people to read. Its documentation lists the addresses, the query parameters and the limits. answers requests sent to web addresses with data, and a long answer comes in pages.
The World Bank’s documentation describes its query parameters: format=json for JSON, per_page for the records on
each page, 50 unless you say otherwise, and page for which page to return. The lesson’s folder holds an answer
recorded from that API: the share of the population using the internet, indicator IT.NET.USER.ZS, for India,
Bangladesh, Nepal and Sri Lanka in five years, asked for 8 records a page. The run reads the first page:
from session import run, version
version()
# R reads api_page.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f api_page.R
""")Output
# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f api_page.R
> # One page of a recorded answer from the World Bank's indicators API:
> # internet users as a share of the population (IT.NET.USER.ZS), CC BY 4.0.
> library(jsonlite)
> page <- fromJSON("api/page-1.json")
> length(page)
[1] 2
> page[[1]][c("page", "pages", "per_page", "total")]
$page
[1] 1
$pages
[1] 3
$per_page
[1] 8
$total
[1] 20
> rows <- page[[2]]
> names(rows)
[1] "indicator" "country" "countryiso3code" "date"
[5] "value" "unit" "obs_status" "decimal"
> names(rows$country)
[1] "id" "value"
> data.frame(country = rows$country$value, year = as.integer(rows$date),
+ internet_pct = rows$value)
country year internet_pct
1 Bangladesh 2025 NA
2 Bangladesh 2024 53.42171
3 Bangladesh 2023 44.50274
4 Bangladesh 2022 41.61650
5 Bangladesh 2021 38.91744
6 India 2025 70.00000
7 India 2024 64.94350
8 India 2023 60.25230
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 api_page_session.py
The answer is a JSON array of two elements. The first says where this page stands: page 1 of 3, 8 records a page,
20 in all. The second is the page’s records, each with the country and the indicator as nested objects, so
rows$country$value holds the country names. The year arrives as text, "2025", and becomes a number with
as.integer(); the value for Bangladesh in 2025 is null in the JSON and NA in R. That is
pagination: Splitting a long answer into pages that a client asks for one at a time, with a page number, an offset or a cursor in each request. The first page usually says how many pages or records there are., and the first element is the key to it: a client asks for page 2 and page 3,
then checks that it holds 20 rows. Step through the loop:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- The first request. The script asks for four countries and five years, 8 records a page. The API answers with page 1 and nothing more.
- The first element. Every page starts with the same small object. Page 1 says page 1, pages 3, per_page 8 and total 20, so there are two more pages to fetch.
- The next page. iterate_with_offset("page") adds page = 2 to the same request, and resp_pages tells it that there are 3 pages in all.
- The last page. Page 3 holds the last 4 records. With page equal to pages there is nothing left to ask for, and req_perform_iterative() stops.
- All the records. resps_data() turns each page into rows and binds them, 8, 8 and 4, into one data frame of 20 rows.
- The count. nrow() is 20, the total the first page promised. A shorter table means a page was lost, and a longer one that records were counted twice.
Where the numbers come from
The data is the World Bank’s World Development Indicators, indicator IT.NET.USER.ZS (Individuals using the
Internet, % of population; source: International Telecommunication Union), licensed under Creative Commons
Attribution 4.0. The three values for 2025 other than India’s were not yet in the data when the answer was recorded.
Checkpoint
Check: Pages of an API answer
httr2 builds the request, follows the pages and waits when told
httr2 builds a request one step at a time: the address, the path, the query, then the policies that govern how it is sent. The first command of the run prints the report’s request; the second replays the three recorded pages through it, so the script runs exactly as it would against the API, without the network:
from session import run, version
version()
# R reads api_request.R, then api_pages.R, as if each line were typed at the console.
run("""
R --vanilla --quiet -f api_request.R
R --vanilla --quiet -f api_pages.R
""")Output
# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f api_request.R
> library(httr2)
> # The request the report sends: four countries, five years, 8 records a page.
> req <- request("https://api.worldbank.org/v2") |>
+ req_url_path_append("country", "IND;BGD;NPL;LKA",
+ "indicator", "IT.NET.USER.ZS") |>
+ req_url_query(format = "json", date = "2021:2025", per_page = 8) |>
+ req_user_agent("internet-use-report (analyst@example.org)") |>
+ req_throttle(capacity = 30, fill_time_s = 60) |>
+ req_retry(max_tries = 3)
> req
<httr2_request>
GET https://api.worldbank.org/v2/country/IND%3BBGD%3BNPL%3BLKA/indicator/IT.NET.USER.ZS?format=json&date=2021%3A2025&per_page=8
Body: empty
Options:
* useragent: "internet-use-report (analyst@example.org)"
Policies:
* throttle_realm : "api.worldbank.org"
* retry_max_tries : 3
* retry_on_failure : FALSE
* retry_failure_threshold: Inf
* retry_failure_timeout : 30
* retry_realm : "api.worldbank.org"
$ R --vanilla --quiet -f api_pages.R
> library(httr2)
> source("api_request.R", echo = FALSE) # builds req, as above
> # Answer each request with the page recorded from the API:
> replay <- function(req) {
+ page <- url_parse(req$url)$query$page
+ file <- sprintf("api/page-%s.json", if (is.null(page)) "1" else page)
+ cat("served", file, "\n")
+ response(status_code = 200, body = readBin(file, "raw", file.size(file)),
+ headers = "Content-Type: application/json;charset=utf-8")
+ }
> pages <- with_mocked_responses(replay, req_perform_iterative(req,
+ next_req = iterate_with_offset("page",
+ resp_pages = function(resp) resp_body_json(resp)[[1]]$pages),
+ max_reqs = Inf, progress = FALSE))
served api/page-1.json
served api/page-2.json
served api/page-3.json
> length(pages)
[1] 3
> internet <- resps_data(pages, function(resp) {
+ rows <- jsonlite::fromJSON(resp_body_string(resp))[[2]]
+ data.frame(country = rows$countryiso3code, year = as.integer(rows$date),
+ internet_pct = rows$value)
+ })
> nrow(internet)
[1] 20
> internet[internet$year >= 2024, ]
country year internet_pct
1 BGD 2025 NA
2 BGD 2024 53.42171
6 IND 2025 70.00000
7 IND 2024 64.94350
11 LKA 2025 NA
12 LKA 2024 54.59000
16 NPL 2025 NA
17 NPL 2024 46.32190
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 api_pages_session.py
In the printed address httr2 has percent-encoded the semicolons between the countries as %3B and the colon of
2021:2025 as %3A, the forms RFC 3986 defines for those characters. The request names its client with
req_user_agent(), the user agent: The User-Agent header of an HTTP request, which names the program sending it. A script that calls an API or fetches pages should name itself and give a way to reach its owner. that tells the API’s owners who is calling and how to
reach them. req_throttle(capacity = 30, fill_time_s = 60) keeps to at most 30 requests a minute, the token bucket
its help page describes. req_perform_iterative() with iterate_with_offset("page") asked for page 2 and page 3
after the first, stopping at the 3 pages the first answer promised, and resps_data() turned the pages into one data
frame of 20 rows. with_mocked_responses() is httr2’s way
of answering requests from your own function, here the recorded files, which is how the run works anywhere.
req_perform_iterative() stops at 20 requests
Its help page gives max_reqs = 20 as the default, so an answer of 24 pages stops after page 20 without an error.
Pass max_reqs = Inf with a way to know the last page, as the example does, and compare the rows with the total.
An API also says no. A server over its rate limit: The most requests a service accepts from one client in a period. A client over it gets the status 429, Too Many Requests, often with a Retry-After header saying how long to wait. answers 429, Too Many Requests
(RFC 6585), and one that is overloaded answers 503, Service Unavailable; either may add a Retry-After header that
says how long to wait (RFC 9110). The run shows what such an answer does to a request without a retry policy:
from session import run, version
version()
# R reads api_errors.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f api_errors.R
""")Output
# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f api_errors.R
> library(httr2)
> # A busy server answers 503 and says when to come back:
> busy <- function(req) response(status_code = 503, headers = "Retry-After: 30")
> err <- tryCatch(
+ with_mocked_responses(busy, req_perform(request("https://api.example.org/v1/visits"))),
+ error = function(e) e)
> class(err)[1]
[1] "httr2_http_503"
> conditionMessage(err)
[1] "HTTP 503 Service Unavailable."
> resp_retry_after(err$resp)
[1] 30
> # The question req_retry() asks of each failed answer, by default:
> transient <- function(resp) resp_status(resp) %in% c(429, 503)
> transient(response(status_code = 429))
[1] TRUE
> transient(response(status_code = 404))
[1] FALSE
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 api_errors_session.py
The 503 stopped the request with an error of class httr2_http_503, and resp_retry_after() read the 30 seconds
from its header. req_retry(max_tries = 3) handles exactly these two statuses by default: its help page says httr2
waits the Retry-After time when there is one, and otherwise a random time between one second and 2 ^ tries
seconds, never more than 60. A mocked response skips that loop, so the next figure follows the help page instead of
a run:
The drawing shows the whole scene; its 9 steps are listed below.
Every step as text
- The request. The script calls req_perform() once. Everything that follows happens inside that call.
- Try 1. httr2 sends the request to the API.
- Busy, with a time. The API answers 503, Service Unavailable, with the header Retry-After 30. 429 and 503 are the statuses httr2 retries by default.
- Waiting as asked. Because the response says how long to wait, httr2 waits exactly that, 30 seconds, before it tries again.
- Try 2. The same request goes out a second time.
- Too many requests. This time the answer is 429, Too Many Requests, with no Retry-After. httr2 then waits a random time between 1 and 2 to the power of the tries so far, 1 to 4 seconds after a second failure, and never more than 60.
- Try 3. The third try is the last that max_tries = 3 allows.
- Success. The API answers 200 with the page, and req_perform() returns it to the script, which never saw the two failures. Had the third try failed too, the call would have ended with the error of that answer, such as HTTP 503 Service Unavailable.
- The result. The script receives one response. req_throttle() spaces out the calls of req_perform(), one for each page of a download; the waits between the tries inside one call come from req_retry().
R running in a web page, as webR does, adds a limit of its own. Browsers let code in a page read a response from
another site only when that site allows it with CORS: Cross-Origin Resource Sharing: the HTTP headers with which a server lets code from other websites read its responses in a browser. Without them a browser blocks the read, which is why R running in a web page cannot call most APIs. headers, MDN explains, and webR’s
networking guide says download.file() works only with such servers, while curl and httr2 need a WebSocket proxy
outside webR. Recorded answers, replayed as above, keep a lesson, a test or a report working wherever it runs.
Checkpoint
Check: Requests with httr2
Keys live in environment variables, never in the script
Many APIs want a key with every request, and a key in a script travels wherever the script goes: into version control, into a printed request, into a screenshot. The run starts with the weak version, then the fix, then a check that tells them apart:
api_key <- "demo-key-not-real-0000"
req <- httr2::request("https://api.example.org/v1/visits") |>
httr2::req_headers(`X-Api-Key` = api_key)
print(req)key <- Sys.getenv("CLINIC_API_KEY")
if (!nzchar(key)) stop("CLINIC_API_KEY is not set: add it to .Renviron")
req <- httr2::request("https://api.example.org/v1/visits") |>
httr2::req_auth_bearer_token(key)
print(req)
cat("The key has", nchar(key), "characters; it was never printed.\n")from session import run, version
version()
# The weak script prints its key; the fixed one reads it from .Renviron. The check passes only on the fix.
run("""
Rscript weak/report.R
cd keys && Rscript report.R
!cd keys && Rscript --no-environ report.R
!Rscript check_keys.R weak/report.R
Rscript check_keys.R keys/report.R
""")Output
# R version 4.6.1 -- "Happy Hop" $ Rscript weak/report.R <httr2_request> GET https://api.example.org/v1/visits Headers: * X-Api-Key: "demo-key-not-real-0000" Body: empty $ cd keys && Rscript report.R <httr2_request> GET https://api.example.org/v1/visits Headers: * Authorization: <REDACTED> Body: empty The key has 22 characters; it was never printed. $ cd keys && Rscript --no-environ report.R Error: CLINIC_API_KEY is not set: add it to .Renviron Execution halted [exit status 1] $ Rscript check_keys.R weak/report.R weak/report.R, line 1: a secret written into the script [exit status 1] $ Rscript check_keys.R keys/report.R No secrets written into keys/report.R
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 keys_session.py
The weak script printed its request, and with it the key, because httr2 prints a header such as X-Api-Key as it is
unless .redact names it. The fixed script read the key with Sys.getenv("CLINIC_API_KEY"), an
environment variable: A named setting that the operating system passes to the programs it starts. R reads one with Sys.getenv() and always gets text. More in “Scripts, projects and file paths that travel” that R set from the .Renviron file of the folder it
started in; R’s Startup help says it looks for .Renviron in the current directory, then in the home directory.
req_auth_bearer_token() sends the key, and the printed request shows <REDACTED> in its place. Started with
--no-environ, the same script stopped with the message it was written to give, instead of calling the API without a
key.
check_keys.R is the check: it fails, with exit status 1, on a script that assigns a quoted value of eight or more
characters to a name containing key, token, secret or password, and passes on the fix. Run it before every
commit, or in the pipeline that tests your project, and add .Renviron to the project’s .gitignore so the file that
holds the key never reaches the repository.
A key that has left your machine
A key that was committed, pushed or pasted anywhere is public, even after you delete the line: the history and every copy keep it. Revoke it with the provider and issue a new one, as OWASP’s guidance on secrets advises for exposed credentials, then fix the code so the new key stays in the environment.
Checkpoint
Check: API keys
Scrape only after the API, the terms and robots.txt
web scraping: Extracting data from web pages written for people, by downloading their HTML and picking out elements such as tables. It comes after an API or a download, and only where the site allows it. reads data out of pages written for people, and it is the last resort: an API or a download gives the same data in a form made for programs, with limits the owner has chosen. When you do scrape, three things decide whether you may and how. Step through them:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- An API first. An API or a download gives the same data in a form made for programs, with documented limits. If one exists, use it and skip scraping altogether.
- robots.txt. With no API, read robots.txt. Only the group that names your user agent applies, else the * group, so rainfall-report may read /tables/ but not /tables/raw/.
- Disallowed. A path the rules disallow is the end of the road for that path, whatever your code could technically fetch. polite_page() stops before it sends a request.
- The terms. robots.txt is not permission. The site's terms of use may forbid copying or automated access, and then the answer is no, or a request to the owner.
- Polite fetching. Name your client and how to reach you in the user agent, wait seconds between requests, and save each page once, so a second run sends nothing.
- The saved copy. Parse from the saved file, as often as you like. html_table() returns text such as 1,012, so convert the numbers and check the rows before you trust them.
The second is a site’s robots.txt: A file at the root of a website that asks automated clients not to fetch certain paths, standardised in RFC 9309. It states a request, not an access control, and only the group for your client applies., the file RFC 9309 standardises. A group of rules
starts with the user agents it names; a client obeys the group that names it, and the * group only when no group
does; within the group, the longest matching rule wins, and allow wins a tie. The RFC adds that the rules are not a
form of access authorization: they are requests, which a careful client honours, and the site’s terms of use can
forbid more. robots.R implements those rules in 25 lines, and the run asks it about three paths of a site made up
for this lesson, then reads the site’s saved table:
# May `agent` fetch `path`, by the rules of robots.txt (RFC 9309)? Only the group
# that names the agent applies, else the * group; the longest matching rule wins,
# and allow wins a tie. Handles * inside a rule and $ at its end.
robots_allows <- function(lines, path, agent) {
lines <- trimws(sub("#.*", "", lines))
field <- tolower(trimws(sub(":.*", "", lines)))
value <- trimws(sub("^[^:]*:", "", lines))
ua <- field == "user-agent"
group <- cumsum(ua & !c(FALSE, head(ua, -1)))
names_in <- tapply(tolower(value[ua]), group[ua], c)
mine <- names(names_in)[vapply(names_in, function(a) tolower(agent) %in% a, TRUE)]
if (!length(mine)) mine <- names(names_in)[vapply(names_in, function(a) "*" %in% a, TRUE)]
best <- -1
allowed <- TRUE
for (i in which(field %in% c("allow", "disallow") & group %in% mine & nzchar(value))) {
rx <- gsub("\\*", ".*", gsub("([.+?^(){}|\\[\\]\\\\])", "\\\\\\1", value[i]))
if (!grepl(paste0("^", rx), path)) next
n <- nchar(value[i], type = "bytes")
if (n > best || (n == best && field[i] == "allow")) {
best <- n
allowed <- field[i] == "allow"
}
}
allowed
}from session import run, version
version()
# R reads scrape_saved.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f scrape_saved.R
""")Output
# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f scrape_saved.R
> source("robots.R")
> rules <- readLines("site/robots.txt")
> for (path in c("/tables/rainfall.html", "/tables/raw/rain.csv", "/private/notes"))
+ cat(sprintf("%-22s rainfall-report: %-5s any other bot: %s\n", path,
+ robots_allows(rules, path, "rainfall-report"),
+ robots_allows(rules, path, "otherbot")))
/tables/rainfall.html rainfall-report: TRUE any other bot: TRUE
/tables/raw/rain.csv rainfall-report: FALSE any other bot: TRUE
/private/notes rainfall-report: TRUE any other bot: FALSE
> library(rvest)
> page <- read_html("site/tables/rainfall.html")
> html_elements(page, "table")
{xml_nodeset (2)}
[1] <table id="rainfall">\n<caption>Seasonal rainfall</caption>\n<thead><tr>\ ...
[2] <table id="contact"><tr><td>Questions: write to the statistics office.</t ...
> rain <- page |> html_element("table#rainfall") |>
+ html_table(na.strings = c("", "—"))
> rain
# A tibble: 3 × 4
Zone `Normal (mm)` `Actual (mm)` Departure
<chr> <chr> <chr> <chr>
1 North 1,012 1,148 13%
2 East 826 702 -15%
3 West 1,104 <NA> <NA>
> rain$`Normal (mm)` <- readr::parse_number(rain$`Normal (mm)`)
> rain$`Normal (mm)`
[1] 1012 826 1104
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 scrape_saved_session.py
rainfall-report may read /tables/ but not /tables/raw/, and it may read /private/notes, because its own group
says nothing about /private/: the * group’s rule binds only clients that no group names. rvest’s read_html()
read the saved page, html_element("table#rainfall") picked the table by its id rather than by position, and
html_table() returned it with the numbers as text, 1,012, which parse_number() converted.
Polite fetching is the rest: name yourself in the user agent, wait between requests, and save every page once, so a
second run sends nothing. polite_page() does all three, and refuses a path before it sends anything:
from session import run, version
version()
# R reads polite.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f polite.R
""")Output
# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f polite.R
> source("robots.R")
> library(httr2)
> # Fetch a page only if robots.txt allows it, say who is asking, wait between
> # requests, and keep a copy so the same page is never fetched twice.
> polite_page <- function(url, rules, agent, cache = "cache", wait = 5) {
+ path <- url_parse(url)$path
+ if (!robots_allows(rules, path, agent)) {
+ stop("robots.txt asks ", agent, " not to fetch ", path, call. = FALSE)
+ }
+ file <- file.path(cache, paste0(gsub("[^A-Za-z0-9]+", "_", path), ".html"))
+ if (!file.exists(file)) {
+ Sys.sleep(wait)
+ resp <- request(url) |> req_user_agent(agent) |> req_perform()
+ dir.create(cache, showWarnings = FALSE)
+ writeLines(resp_body_string(resp), file)
+ }
+ rvest::read_html(file)
+ }
> # Replay the site from its saved copy, and count the requests that reach it:
> sent <- 0
> site <- function(req) {
+ sent <<- sent + 1
+ cat("request", sent, "for", url_parse(req$url)$path, "from", req$options$useragent, "\n")
+ response(status_code = 200, headers = "Content-Type: text/html",
+ body = readBin("site/tables/rainfall.html", "raw", 4096))
+ }
> rules <- readLines("site/robots.txt")
> url <- "https://stats.example.org/tables/rainfall.html"
> with_mocked_responses(site, {
+ a <- polite_page(url, rules, "rainfall-report", wait = 0)
+ b <- polite_page(url, rules, "rainfall-report", wait = 0)
+ try(polite_page("https://stats.example.org/tables/raw/rain.csv", rules,
+ "rainfall-report", wait = 0))
+ })
request 1 for /tables/rainfall.html from rainfall-report
Error : robots.txt asks rainfall-report not to fetch /tables/raw/rain.csv
> sent
[1] 1
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 polite_session.py
Two calls for the same allowed page sent one request, with the user agent rainfall-report, and the disallowed
/tables/raw/ stopped with an error before any request; the replay counted one in all. Against a real site, keep
wait at several seconds.
Checkpoint
Check: Responsible scraping
Practise: predict, then check
Eight lines go back over the lesson’s JSON and its robots.txt rules. Predict each result before you look at the run.
Before you look
Predict what fromJSON() makes of an array that mixes numbers and one string, and of two records with different
names; what toJSON() writes for a list of one value with auto_unbox = TRUE and for a one-row data frame holding
NA; how many records the last recorded page holds; and three answers of robots_allows(). Then type the lines into
your own R.
from session import run, version
version()
# R reads practice.R as if each line were typed at the console.
run("""
R --vanilla --quiet -f practice.R
""")Output
# R version 4.6.1 -- "Happy Hop"
$ R --vanilla --quiet -f practice.R
> library(jsonlite)
> source("robots.R")
> # Predict each result before you look.
> fromJSON('[1, "2", 3]')
[1] "1" "2" "3"
> fromJSON('[{"a": 1}, {"b": 2}]')
a b
1 1 NA
2 NA 2
> toJSON(list(n = 1), auto_unbox = TRUE)
{"n":1}
> toJSON(data.frame(x = NA_real_))
[{}]
> nrow(fromJSON("api/page-3.json")[[2]])
[1] 4
> robots_allows(c("User-agent: *", "Disallow: /"), "/tables/a.html", "anybot")
[1] FALSE
> robots_allows(c("User-agent: *", "Disallow:"), "/tables/a.html", "anybot")
[1] TRUE
> robots_allows(c("User-agent: *", "Disallow: /tables/", "Allow: /tables/a"),
+ "/tables/a.html", "anybot")
[1] TRUE
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 practice_session.py
The mixed array became text, the two records became two rows with NA where a name was missing, the single value was
unboxed, the record with only NA was written as {}, the last page held 4 records, Disallow: / closed every path,
an empty Disallow: closed none, and the longer Allow beat the shorter Disallow.
Then the questions, one at a time, the idea you have shown least first, each explained when you check it:
Practice that adapts to you
Practise this lesson
17 questions about 6 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
Warm-up: what does jsonlite’s fromJSON() return for an array of JSON objects, and how does that differ from what httr2’s resp_body_json() returns?
fromJSON() simplifies: an array of objects that share their names becomes a data frame with one row per object, a
missing name or a null becomes NA, a nested object becomes a data frame column (or flat columns with
flatten = TRUE) and a nested array becomes a list column. resp_body_json() does not simplify by default: it returns
the raw shape, a list with one list per object, and a null stays NULL. So code that builds rows from
resp_body_json() must turn each NULL into NA itself, or a missing value shortens a row and shifts its columns.
When you want a data frame from an httr2 response, pass resp_body_string(resp) to fromJSON(), as the lesson’s
example does, and check the types with str() before you use them.
Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.
-
Mid to senior
Your script pages through an API 100 records at a time with req_perform_iterative() and iterate_with_offset(). The first page says total 2,345, but the table has 2,000 rows and no error appeared. What happened, and how do you make the download complete and checked?
-
Fresher to mid
A teammate's analysis calls a paid weather API with the key written into the script, and prints each request while debugging. The repository is about to become public. What are the risks, and what exactly do you change before it goes public?
Key takeaways
fromJSON()maps arrays of values to vectors, objects to named lists and arrays of records to data frames, withnullasNA, nested objects as data frame columns and nested arrays as list columns.toJSON()writes rows, leavesNAfields out, rounds to four digits and boxes single values, so setna,digitsandauto_unboxto what the receiving program expects.- An API answers in pages: read the page facts it sends, request every page, and check the rows against the total.
- httr2 builds requests with a user agent, a throttle and a retry policy, follows pages with
req_perform_iterative()(passmax_reqs), and replays recorded answers withwith_mocked_responses(). - Keep keys in environment variables set by
.Renviron, out of version control, send them withreq_auth_bearer_token()or a redacted header, and run a check that fails on any script that holds one. - Scrape only when there is no API, the terms allow it and robots.txt does too, and then politely: a named user agent, waits between requests and one saved copy of each page.
Terms in this lesson
- CORS
- Cross-Origin Resource Sharing: the HTTP headers with which a server lets code from other websites read its responses in a browser. Without them a browser blocks the read, which is why R running in a web page cannot call most APIs.
- JSON
- JavaScript Object Notation, a text format for data built from objects in braces, arrays in brackets, strings, numbers, true, false and null (RFC 8259). Most web APIs answer in it.
- pagination
- Splitting a long answer into pages that a client asks for one at a time, with a page number, an offset or a cursor in each request. The first page usually says how many pages or records there are.
- rate limit
- The most requests a service accepts from one client in a period. A client over it gets the status 429, Too Many Requests, often with a Retry-After header saying how long to wait.
- robots.txt
- A file at the root of a website that asks automated clients not to fetch certain paths, standardised in RFC 9309. It states a request, not an access control, and only the group for your client applies.
- user agent
- The User-Agent header of an HTTP request, which names the program sending it. A script that calls an API or fetches pages should name itself and give a way to reach its owner.
- web API
- A service that answers requests sent to web addresses with data, usually JSON, rather than pages for people to read. Its documentation lists the addresses, the query parameters and the limits.
- web scraping
- Extracting data from web pages written for people, by downloading their HTML and picking out elements such as tables. It comes after an API or a download, and only where the site allows it.
References
- jsonlite: A Simple and Robust JSON Parser and Generator for R (CRAN)
- Getting started with JSON and jsonlite (jsonlite authors (CRAN))
- The jsonlite Package: A Practical and Consistent Mapping Between JSON Data and R Objects (arXiv (Jeroen Ooms))
- RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Format (IETF)
- httr2: Perform HTTP requests and process the responses (httr2 authors (Posit))
- httr2: Automatically retry a request on failure (req_retry) (httr2 authors (Posit))
- httr2: Rate limit a request by automatically adding a delay (req_throttle) (httr2 authors (Posit))
- httr2: Perform requests iteratively (req_perform_iterative) (httr2 authors (Posit))
- httr2: Iteration helpers (iterate_with_offset) (httr2 authors (Posit))
- httr2: Temporarily mock requests (with_mocked_responses) (httr2 authors (Posit))
- httr2: Modify request headers (req_headers) (httr2 authors (Posit))
- httr2: Wrapping APIs (httr2 authors (Posit))
- rvest: Easily Harvest (Scrape) Web Pages (rvest authors (Posit))
- rvest: Parse an html table into a data frame (html_table) (rvest authors (Posit))
- webR: Networking (webR authors (Posit))
- Cross-Origin Resource Sharing (CORS) (MDN Web Docs (Mozilla))
- RFC 9309: Robots Exclusion Protocol (IETF)
- RFC 6585: Additional HTTP Status Codes (429 Too Many Requests) (IETF)
- RFC 9110: HTTP Semantics (503 and Retry-After) (IETF)
- Initialization at Start of an R Session (Startup, .Renviron) (The R Core Team)
- Secrets Management Cheat Sheet (OWASP Foundation)
- API Basic Call Structures (The World Bank)
- World Development Indicators (licence) (The World Bank)
- Individuals using the Internet (% of population), IT.NET.USER.ZS (The World Bank)
- RFC 3986: Uniform Resource Identifier (URI): Generic Syntax (percent-encoding) (IETF)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress