Remote work: ssh, rsync and curl in scripts

ssh web1 wc -l "$f" looks like one command, yet with f set to sales 2027.csv the far end looks for two files, sales and 2027.csv, and fails with status 1. Scripts that reach other computers break in ways a local script cannot: a second shell re-reads your words, a copy re-sends 260,000 bytes where 700 would do, a host that stops answering holds the script until someone notices, and curl exits 0 on an HTTP 404.

Every one of those cases runs in this lesson’s sessions on one Mac, against servers on 127.0.0.1 and a local stand-in for the far end of ssh: the two parses of an ssh command and the two ways to protect your arguments, the options that make ssh give up in seconds, what rsync decides to send and how little of a file it sends, the statuses curl and jq leave for a script to read, and a port check with nothing but Bash.

ssh sends one string, and the far end’s shell parses it again

Your shell splits a command line into words and removes the quotes, so ssh receives tidy arguments. It does not send them as a list. The ssh(1) manual says that any arguments after the command are appended to it, separated by spaces, before the line goes to the server. On the far side, sshd(8) runs that line through the account’s login shell with its -c option, and that shell splits, expands and runs it like any line it reads. Every word of a remote commandremote command: A command line given to ssh after the host name. ssh joins its words with spaces and sends one string, which the far account's login shell parses again before running it, so its quoting must survive two shells. is parsed twice: once by your shell and once by a shell you never see.

This Mac runs no SSH server, so the session plays the far end with a small function, far_end, that does exactly those two things: it joins its arguments with spaces, prints the result and hands it to a second Bash with bash -c. Everything else is a real run of Bash 5.3.20:

"""ssh joins its words into one string and the far end's shell parses that string again.

This Mac runs no SSH server, so the function far_end plays the far side: it joins its arguments with spaces, which is
what ssh(1) says the client does with a command and its arguments, and hands that one string to a second Bash, which
is what sshd(8) does with it (it runs the user's shell with -c). Everything else is real Bash 5.3.20.
"""
from session import run, write

write("sales 2027.csv", "north,120\nsouth,95\neast,143\n")
write(
    "count.sh",
    "#!/bin/bash\n"
    "# count.sh: sent on standard input, so the far end's login shell never parses its text\n"
    "printf 'counting %s\\n' \"$1\"\n"
    "wc -l < \"$1\"\n",
)

run(r"""
# Here, the quotes keep the name whole: wc gets -l and one file name.
f='sales 2027.csv'
wc -l "$f"
# far_end stands in for ssh plus the remote shell: join the words with spaces, then parse the string again.
far_end() { local IFS=' '; printf 'far end runs: %s\n' "$*"; bash -c "$*"; }
!far_end wc -l "$f"
printf -v q '%q' "$f"
echo "$q"
far_end wc -l "$q"
# A semicolon in a value becomes a second command on the far end.
g='q3.csv; touch MARKER'
far_end wc -l "$g"
ls
rm MARKER
printf -v qg '%q' "$g"
!far_end wc -l "$qg"
ls
# Sending the script on standard input keeps its text away from the far shell, but not the words after bash -s.
cat count.sh
!far_end bash -s -- "$f" < count.sh
far_end bash -s -- "$q" < count.sh
""")

# Here, the quotes keep the name whole: wc gets -l and one file name.
$ f='sales 2027.csv'
$ wc -l "$f"
       3 sales 2027.csv
# far_end stands in for ssh plus the remote shell: join the words with spaces, then parse the string again.
$ far_end() { local IFS=' '; printf 'far end runs: %s\n' "$*"; bash -c "$*"; }
$ far_end wc -l "$f"
far end runs: wc -l sales 2027.csv
wc: sales: open: No such file or directory
wc: 2027.csv: open: No such file or directory
       0 total
[exit status 1]
$ printf -v q '%q' "$f"
$ echo "$q"
sales\ 2027.csv
$ far_end wc -l "$q"
far end runs: wc -l sales\ 2027.csv
       3 sales 2027.csv
# A semicolon in a value becomes a second command on the far end.
$ g='q3.csv; touch MARKER'
$ far_end wc -l "$g"
far end runs: wc -l q3.csv; touch MARKER
wc: q3.csv: open: No such file or directory
$ ls
MARKER
count.sh
sales 2027.csv
$ rm MARKER
$ printf -v qg '%q' "$g"
$ far_end wc -l "$qg"
far end runs: wc -l q3.csv\;\ touch\ MARKER
wc: q3.csv; touch MARKER: open: No such file or directory
[exit status 1]
$ ls
count.sh
sales 2027.csv
# Sending the script on standard input keeps its text away from the far shell, but not the words after bash -s.
$ cat count.sh
#!/bin/bash
# count.sh: sent on standard input, so the far end's login shell never parses its text
printf 'counting %s\n' "$1"
wc -l < "$1"
$ far_end bash -s -- "$f" < count.sh
far end runs: bash -s -- sales 2027.csv
counting sales
bash: line 4: sales: No such file or directory
[exit status 1]
$ far_end bash -s -- "$q" < count.sh
far end runs: bash -s -- sales\ 2027.csv
counting sales 2027.csv
       3

Locally, wc -l "$f" counted 3 lines. Through far_end, the same words became the string wc -l sales 2027.csv, and the second shell gave wc two names that do not exist. The value q3.csv; touch MARKER did worse: the semicolon ended the first command on the far end, touch MARKER ran as a second one, and the next ls listed MARKER. That is command injectioncommand injection: A bug where user text is read as shell: a value such as x; touch MARKER arrives through eval, sh -c or unquoted expansion and runs its own commands. The fix is to pass the value as one argument, never splice it into a command string. More in “Injection, eval and untrusted input”, caused by nothing more than a value that holds shell syntax. Step through the first case and the two fixes the session ran:

The two fixes protect different things, so a script often uses both:

• Quote each value for the far shell. printf -v q '%q' "$f" stored sales\ 2027.csv, a form the second shell reads back as the original value; this is a shell escapeshell escape: Turning a value into text the shell will read back as the same value, with every special character made safe. Bash's printf %q prints a value in this form; the Bash manual says the result "can be reused as shell input". More in “Injection, eval and untrusted input”. With it, wc got one name and counted 3, and the hostile value became one odd file name that wc could not open, with no MARKER made.
• Send the script on standard input. ssh web1 bash -s -- "$q" < count.sh carries the text of count.sh as data, which no shell on the far end parses as a command line; bash -s reads it from its input and runs it with the words after -- as $1, $2 and so on. Those words still go through the second parse: with "$f" the script received $1 as sales, and only "$q" gave it the whole name.

The rules for turning data into shell input safely, and why eval is the same trap, are in the lesson on eval and untrusted input. Keep the ssh options themselves in a command arraycommand array: An indexed array that holds a command name and its arguments, one per element, run as "${cmd[@]}" so that each element reaches the command as exactly one word, spaces and quote characters included. More in “Lists done right: arrays, mapfile and read -a”, as in arrays for commands, so each option reaches ssh as one word.

ssh in a script must fail fast, and 255 means ssh itself failed

At a terminal, ssh asks questions: a password, a passphrase, whether to trust a host it has never seen. A script run by cron or CI has nobody to answer them, and a host that has stopped responding can hold its connection open long after anyone would wait. A script therefore logs in by public-key authenticationpublic-key authentication: Logging in by proving you hold a private key whose public half the server lists for the account, in ~/.ssh/authorized_keys for OpenSSH. No password travels, so a script can log in without a prompt.: ssh(1) says the far account’s ~/.ssh/authorized_keys lists the public keys allowed to log in, so the private half never leaves the machine that runs the script. Then a handful of options turn every question into a quick failure:

Batch modebatch mode: The ssh option BatchMode yes, which turns off every question ssh would ask a person (a password, a passphrase, whether to trust a new host key), so a script gets an error and status 255 instead of a prompt. matters most of all, because it turns “waiting for someone to type” into an error a script can see. The host keys themselves live in a known hostsknown hosts file: A file of host names and the public host keys ssh expects from them, ~/.ssh/known_hosts by default. ssh checks each server's key against it and refuses to connect, or warns, when the key differs. file; a script that names its own with UserKnownHostsFile checks every host against keys it ships, instead of against whatever a person’s account has collected. The order of a config file matters too. ssh_config(5) says the first value found for each option is the one used, so host-specific blocks go first and the defaults for Host * go last. The session writes such a file, asks ssh -G what a connection would use without connecting, and then connects only to 127.0.0.1: once to a port nothing listens on, and once to a port where a server accepts the connection and then says nothing, as a host that has hung would. The system picks those ports for each run; the session prints them as CLOSED_PORT and SILENT_PORT:

"""ssh options that make a script fail within seconds instead of waiting, and the status 255.

The ssh client really runs here (OpenSSH 10.3p1), but only with the session's own files: -F names the session's
ssh_config (so neither ~/.ssh/config nor /etc/ssh/ssh_config is read), known hosts are a file of the session and
/dev/null, and IdentityFile=none with IdentityAgent=none offers no key at all. IdentityFile=none matters: ssh takes ~
from the account database, not from $HOME, and loads its default keys before it connects, so without it a connection
attempt would look for keys in the real home folder. It only ever connects to 127.0.0.1: to a port nothing listens on, and to a
port where servers.silent() accepts the connection and then says nothing, as a host that has hung would. The ports
are chosen by the system for each run and printed as CLOSED_PORT and SILENT_PORT.
"""
from session import env, mask, run, write
import servers

closed_port = servers.closed()
silent_port = servers.silent()
env("CLOSED_PORT", closed_port)
env("SILENT_PORT", silent_port)
mask(closed_port, "CLOSED_PORT")
mask(silent_port, "SILENT_PORT")

write(
    "ssh_config",
    "# Host blocks first, defaults last: for each option, the first value ssh finds wins.\n"
    "Host web1\n"
    "    HostName 127.0.0.1\n"
    "    User deploy\n"
    "    ConnectTimeout 3\n"
    "\n"
    "Host web*\n"
    "    User admin\n"
    "    ServerAliveInterval 15\n"
    "    ServerAliveCountMax 3\n"
    "\n"
    "Host *\n"
    "    BatchMode yes\n"
    "    ConnectTimeout 10\n"
    "    StrictHostKeyChecking yes\n",
)

run(r"""
cat ssh_config
opts=(-F "$PWD/ssh_config" -o BatchMode=yes -o IdentitiesOnly=yes -o IdentityFile=none -o IdentityAgent=none -o UserKnownHostsFile="$PWD/known_hosts" -o GlobalKnownHostsFile=/dev/null)
# ssh -G prints the settings a connection would use, without connecting.
ssh "${opts[@]}" -G web1 true | grep -E '^(user|hostname|batchmode|connecttimeout|serveraliveinterval|serveralivecountmax|stricthostkeychecking) '
ssh "${opts[@]}" -G web2 true | grep -E '^(user|hostname|connecttimeout) '
# Nothing listens on CLOSED_PORT, so the connection is refused at once.
!ssh "${opts[@]}" -p "$CLOSED_PORT" web1 uptime
# SILENT_PORT accepts the connection and then never answers.
start=$SECONDS
!ssh "${opts[@]}" -o ConnectTimeout=2 -p "$SILENT_PORT" web1 uptime
(( SECONDS - start < 4 )) && echo "gave up in under 4 seconds"
# 255 is ssh's own failure; any other status came from the remote command.
on() { ssh "${opts[@]}" "$@"; local rc=$?; if (( rc == 255 )); then echo "ssh failed, so no command ran" >&2; fi; return "$rc"; }
!on -p "$CLOSED_PORT" web1 uptime
ls
""")

$ cat ssh_config
# Host blocks first, defaults last: for each option, the first value ssh finds wins.
Host web1
    HostName 127.0.0.1
    User deploy
    ConnectTimeout 3

Host web*
    User admin
    ServerAliveInterval 15
    ServerAliveCountMax 3

Host *
    BatchMode yes
    ConnectTimeout 10
    StrictHostKeyChecking yes
$ opts=(-F "$PWD/ssh_config" -o BatchMode=yes -o IdentitiesOnly=yes -o IdentityFile=none -o IdentityAgent=none -o UserKnownHostsFile="$PWD/known_hosts" -o GlobalKnownHostsFile=/dev/null)
# ssh -G prints the settings a connection would use, without connecting.
$ ssh "${opts[@]}" -G web1 true | grep -E '^(user|hostname|batchmode|connecttimeout|serveraliveinterval|serveralivecountmax|stricthostkeychecking) '
user deploy
hostname 127.0.0.1
batchmode yes
stricthostkeychecking true
serveralivecountmax 3
serveraliveinterval 15
connecttimeout 3
$ ssh "${opts[@]}" -G web2 true | grep -E '^(user|hostname|connecttimeout) '
user admin
hostname web2
connecttimeout 10
# Nothing listens on CLOSED_PORT, so the connection is refused at once.
$ ssh "${opts[@]}" -p "$CLOSED_PORT" web1 uptime
ssh: connect to host 127.0.0.1 port CLOSED_PORT: Connection refused
[exit status 255]
# SILENT_PORT accepts the connection and then never answers.
$ start=$SECONDS
$ ssh "${opts[@]}" -o ConnectTimeout=2 -p "$SILENT_PORT" web1 uptime
Connection timed out during banner exchange
Connection to 127.0.0.1 port SILENT_PORT timed out
[exit status 255]
$ (( SECONDS - start < 4 )) && echo "gave up in under 4 seconds"
gave up in under 4 seconds
# 255 is ssh's own failure; any other status came from the remote command.
$ on() { ssh "${opts[@]}" "$@"; local rc=$?; if (( rc == 255 )); then echo "ssh failed, so no command ran" >&2; fi; return "$rc"; }
$ on -p "$CLOSED_PORT" web1 uptime
ssh: connect to host 127.0.0.1 port CLOSED_PORT: Connection refused
ssh failed, so no command ran
[exit status 255]
$ ls
ssh_config

web1 matched the web1 block first, so it got user deploy and connecttimeout 3, not the admin and 10 that appear later in the file, while serveraliveinterval 15 came from web* and batchmode yes applied to every host. web2 matched only the later blocks. The refused connection failed at once; the silent one ended with Connection timed out during banner exchange because -o ConnectTimeout=2 on the command line came before the file’s 3. Both ended with status 255, and ls showed that no known hosts file had been written.

The opts array keeps the session away from this computer’s own ssh setup. -F names the session’s file in place of the account’s own, and ssh(1) adds that the system-wide file is then ignored; the two known hosts options name a file of the session and /dev/null; and IdentityFile=none with IdentityAgent=none offers no key at all, since nothing here logs in. A real job names its one key instead, -i deploy_key -o IdentitiesOnly=yes, which ssh_config(5) says keeps ssh to the configured identity even when an agent offers others.

That 255 is the number to test. ssh(1) keeps 255 for its own errors and otherwise passes on the remote command’s status, so the on function could say “no command ran” with certainty. A remote command that itself exits 255 looks the same, which is one more reason to keep remote commands small and their statuses ordinary.

scp, sftp and rsync: three ways to copy files over SSH

All three ride on an SSH connection, so the options of the last section apply to each of them, and each accepts -o options or an ssh command of its own. They differ in what they send and in what they can do besides sending.

scp copies the files you name, in full. Since OpenSSH 9.0 it uses the SFTP protocol by default, and the release notes say why that matters to a script: the old scp/rcp protocol expanded remote file names through the far end’s shell, so names with shell characters needed a second layer of quoting. scp -O still selects the old protocol for servers without SFTP, and scp(1) warns that it then needs careful quoting for that shell.

sftp -b batchfile runs a list of file commands without anyone at the keyboard, and sftp(1) says it aborts when get, put, rm, mkdir or a few other commands fail, so a script gets a status it can trust. It can also continue a partial file with reget, reput or -a.

rsync compares before it copies. It starts a second rsync on the far end through the remote shell, and rsync(1) says plainly that rsync must be installed on both machines. Answer the three questions of the table for your own copy:

In short: rsync for anything a script copies again and again, sftp -b when the far account allows SFTP and nothing else, and scp for a one-off copy of a few files.

rsync’s quick check decides what to send, and the delta decides how much

The session copies a small site from site/ to live/ with -a (archive mode: recursive, keeping links, permissions and times) and -i, which prints one line per item that changes. Both folders are local, but the engine is the one that runs over ssh. On this Mac, /usr/bin/rsync is openrsync, as its first two lines say:

"""What rsync decides to send (the quick check) and how much of a file it sends (the delta).

rsync here is macOS's /usr/bin/rsync, which is openrsync (protocol 29); both copies are local folders, the same
engine that runs over ssh. rsync copies whole files between two local folders by default (rsync(1), --whole-file),
so the last runs add --no-W to make it use the delta-transfer algorithm, as it does between two computers, and -c so
the file is compared by its contents whatever its time says.
"""
from session import run, write

write("site/index.html", "<h1>Orders</h1>\n")
write("site/app.conf", "max_conn=100\n")
write("site/css/main.css", "body { margin: 0 }\n")

run(r"""
rsync --version | head -n 2
rsync -a -i site/ live/
rsync -a -i site/ live/
# A real change with the same size and the same time: the quick check sees nothing to send.
printf 'max_conn=900\n' > site/app.conf
touch -r live/app.conf site/app.conf
rsync -a -i site/ live/
cat live/app.conf
rsync -a -i -c site/ live/
cat live/app.conf
# A 260,000-byte log, copied once; then one byte changes in its middle.
mkdir logs
printf 'line %05d abcdefghijabcdefghijabcdefghijabcdefghij\n' $(seq 0 4999) > logs/app.log
wc -c < logs/app.log
rsync -a logs/ copy/
printf 'X' | dd of=logs/app.log bs=1 seek=130000 conv=notrunc 2>/dev/null
rsync -a -c --stats logs/ copy/ | grep -E '^(Matched|Unmatched) data'
printf 'Y' | dd of=logs/app.log bs=1 seek=200000 conv=notrunc 2>/dev/null
rsync -a -c --no-W --stats logs/ copy/ | grep -E '^(Matched|Unmatched) data'
cmp logs/app.log copy/app.log && echo identical
""")

$ rsync --version | head -n 2
openrsync: protocol version 29
rsync version 2.6.9 compatible
$ rsync -a -i site/ live/
cd+++++++ ./
>f+++++++ app.conf
>f+++++++ index.html
cd+++++++ css/
>f+++++++ css/main.css
$ rsync -a -i site/ live/
# A real change with the same size and the same time: the quick check sees nothing to send.
$ printf 'max_conn=900\n' > site/app.conf
$ touch -r live/app.conf site/app.conf
$ rsync -a -i site/ live/
$ cat live/app.conf
max_conn=100
$ rsync -a -i -c site/ live/
>fc...... app.conf
$ cat live/app.conf
max_conn=900
# A 260,000-byte log, copied once; then one byte changes in its middle.
$ mkdir logs
$ printf 'line %05d abcdefghijabcdefghijabcdefghijabcdefghij\n' $(seq 0 4999) > logs/app.log
$ wc -c < logs/app.log
  260000
$ rsync -a logs/ copy/
$ printf 'X' | dd of=logs/app.log bs=1 seek=130000 conv=notrunc 2>/dev/null
$ rsync -a -c --stats logs/ copy/ | grep -E '^(Matched|Unmatched) data'
Unmatched data: 260000 B
Matched data: 0 B
$ printf 'Y' | dd of=logs/app.log bs=1 seek=200000 conv=notrunc 2>/dev/null
$ rsync -a -c --no-W --stats logs/ copy/ | grep -E '^(Matched|Unmatched) data'
Unmatched data: 700 B
Matched data: 259300 B
$ cmp logs/app.log copy/app.log && echo identical
identical

The first run created everything, so each line begins >f+++++++ (a file received, every attribute new) or cd+++++++ (a folder created). The second run printed nothing at all, because nothing differed. That silence comes from the quick check: rsync(1) says rsync looks for files that changed in size or in modification time, and skips the rest without reading them.

The next lines show where the quick check is blind. app.conf changed from max_conn=100 to max_conn=900, the same 13 bytes, and touch -r gave it the time of the copy already in live/. rsync sent nothing, and live/app.conf still said 100. With -c, rsync compared a checksum of every file whose size matched, found the difference, and printed >fc......, where c means the checksum differs. The price is that both sides read every byte of every file they compare, so -c is for files that tools rewrite without changing their size or time.

Once rsync has decided to send a file, the delta-transfer algorithmdelta-transfer algorithm: How rsync updates a file the far end already has: the receiver sends checksums of the blocks of its copy, and the sender sends only the bytes that match none of them, plus block numbers for the rest. decides how much of it to send. The rsync technical report describes it: the receiver splits its old copy into blocks and sends a checksum of each, and the sender sends literal bytes only where no block matches. The session’s 260,000-byte log shows both modes. Between two local folders rsync copies whole files by default (rsync(1), --whole-file): there is no slow link to save, and working out the delta costs extra reading. So the first changed byte cost 260,000 unmatched bytes. With --no-W, as between two computers, one changed byte cost 700, and 259,300 bytes were matched blocks the receiver already had:

How to mirror a folder with --delete, and why site/ and site copy different things, is in the lesson on archives, checksums and backups.

rsync over ssh needs rsync at both ends, and its status says what broke

Over the network, rsync reaches the far end with -e, which names the remote shell and its options as one string. rsync(1) says the string may carry arguments, separated by spaces, and that rsync splits it itself, so it is built from plain words here rather than from an array. The RSYNC_RSH variable does the same job for every rsync a script runs. The session points that ssh at a closed port on 127.0.0.1, then copies a missing folder and a real one:

"""rsync over ssh in a script: the -e command and the exit statuses a script must read.

rsync here is macOS's /usr/bin/rsync, which is openrsync (protocol 29). The ssh it starts gets the session's own
ssh_config and options, offers no key (IdentityFile=none, IdentityAgent=none) and only ever connects to 127.0.0.1, to
a port nothing listens on (shown as CLOSED_PORT), so nothing is sent anywhere. rsync's process number in its messages changes from run to run and is shown as PID.
"""
from session import env, mask, mask_re, run, write
import servers

closed_port = servers.closed()
env("CLOSED_PORT", closed_port)
mask(closed_port, "CLOSED_PORT")
mask_re(r"rsync\(\d+\)", "rsync(PID)")

write("ssh_config", "Host web1\n    HostName 127.0.0.1\n    User deploy\n\nHost *\n    BatchMode yes\n    ConnectTimeout 5\n")
write("site/index.html", "<h1>Orders</h1>\n")

run(r"""
# rsync splits the -e string at spaces itself, so it is one argument here, built from plain words.
rsh="ssh -F $PWD/ssh_config -o BatchMode=yes -o IdentitiesOnly=yes -o IdentityFile=none -o IdentityAgent=none -o UserKnownHostsFile=$PWD/known_hosts -o GlobalKnownHostsFile=/dev/null -p $CLOSED_PORT"
!rsync -a -e "$rsh" site/ web1:/srv/site/
# A source that is not there: nothing to send, and a status of its own.
!rsync -a reports/ backup/
rsync -a site/ backup/; echo "status $?"
""")

# rsync splits the -e string at spaces itself, so it is one argument here, built from plain words.
$ rsh="ssh -F $PWD/ssh_config -o BatchMode=yes -o IdentitiesOnly=yes -o IdentityFile=none -o IdentityAgent=none -o UserKnownHostsFile=$PWD/known_hosts -o GlobalKnownHostsFile=/dev/null -p $CLOSED_PORT"
$ rsync -a -e "$rsh" site/ web1:/srv/site/
ssh: connect to host 127.0.0.1 port CLOSED_PORT: Connection refused
rsync(PID): error: unexpected end of file
[exit status 255]
# A source that is not there: nothing to send, and a status of its own.
$ rsync -a reports/ backup/
rsync(PID): error: reports/: (l)stat: No such file or directory
[exit status 23]
$ rsync -a site/ backup/; echo "status $?"
status 0

When ssh could not connect, rsync printed ssh’s own message, then unexpected end of file, and ended with 255, the status ssh gives for its own failures. The missing source ended with 23, which rsync(1)’s EXIT VALUES table calls a partial transfer due to an error; 24 in the same table means source files vanished during the copy, which happens when a log is rotated mid-copy. One detail is worth remembering: openrsync’s own manual page lists only 0, 1 and 2, yet it returned 23 and 255 here. Treat every status but 0 as a failed copy, and log the number.

Three more options belong in a script that copies over a network. -z compresses file data as it travels, which helps on a slow link but costs processor time at both ends, so it earns its place only when the link is the slow part; data that is already compressed, such as a .gz or .jpg file, barely shrinks, so the work is wasted on it. --partial keeps a partly sent file when the transfer is cut, which rsync(1) says makes the next transfer of the rest of that file much faster. And -n with -i (a dry run) prints the same itemized lines a real run would, without changing anything, which rsync(1) says is the point of a dry run: read it before any run that deletes.

curl in a script must be told that an HTTP error is a failure

Every HTTP answer starts with an HTTP status codeHTTP status code: The three-digit number at the start of every HTTP answer. Its first digit gives the class: 1xx informational, 2xx success, 3xx redirection, 4xx an error in the request, 5xx an error at the server., and RFC 9110 sorts them by their first digit: 1xx informational, 2xx success, 3xx redirection, 4xx an error in the request, 5xx an error at the server. To curl, though, a 404 page that arrived intact is a successful transfer. The session runs it against a small HTTP server on 127.0.0.1, which starts with the example and stops with it (its port prints as WEB_PORT, and the milliseconds curl reports as N):

"""How curl reports what went wrong: HTTP errors, a redirect, a stalled answer, a refused connection, and a retry.

curl is macOS's /usr/bin/curl 8.7.1. It talks only to servers.web(), a small HTTP server that this example starts on
127.0.0.1 and stops when it ends; API holds its address. The port the system chose and the milliseconds curl reports
change from run to run, so they are shown as WEB_PORT, CLOSED_PORT and N.
"""
from session import env, mask, mask_re, run
import servers

ok = '{"status":"ok","version":"2.4.1"}\n'
web_port = servers.web({
    "/health": [(200, ok)],
    "/latest": [(302, "", "text/plain", 0, {"Location": "/reports/v2.json"})],
    "/reports/v2.json": [(200, '{"report":"v2","rows":3}\n')],
    "/slow": [(200, ok, "application/json", 3)],
    "/busy": [(503, '{"error":"warming up"}\n'), (503, '{"error":"warming up"}\n'), (200, ok)],
})
closed_port = servers.closed()
env("API", f"http://127.0.0.1:{web_port}")
env("CLOSED_PORT", closed_port)
mask(web_port, "WEB_PORT")
mask(closed_port, "CLOSED_PORT")
mask_re(r"after \d+ milliseconds", "after N milliseconds")
mask_re(r"after \d+ ms", "after N ms")

run(r"""
curl -sS "$API/reports/q3"; echo "curl status: $?"
!curl -fsS "$API/reports/q3"
!curl -sS --fail-with-body "$API/reports/q3"
curl -sS -o /dev/null -w '%{http_code}\n' "$API/reports/q3"
# A redirect is not an error: without -L, curl saves the empty redirect answer and succeeds.
curl -fsS -o latest.json "$API/latest"; echo "status $?"
test -s latest.json || echo "latest.json is empty"
curl -fsSL -o latest.json "$API/latest"; cat latest.json
!curl -fsS --max-time 1 "$API/slow"
!curl -fsS "http://127.0.0.1:$CLOSED_PORT/health"
curl -fsS --retry 3 --retry-delay 1 "$API/busy"
""")

$ curl -sS "$API/reports/q3"; echo "curl status: $?"
{"error":"no such path"}
curl status: 0
$ curl -fsS "$API/reports/q3"
curl: (22) The requested URL returned error: 404
[exit status 22]
$ curl -sS --fail-with-body "$API/reports/q3"
curl: (22) The requested URL returned error: 404
{"error":"no such path"}
[exit status 22]
$ curl -sS -o /dev/null -w '%{http_code}\n' "$API/reports/q3"
404
# A redirect is not an error: without -L, curl saves the empty redirect answer and succeeds.
$ curl -fsS -o latest.json "$API/latest"; echo "status $?"
status 0
$ test -s latest.json || echo "latest.json is empty"
latest.json is empty
$ curl -fsSL -o latest.json "$API/latest"; cat latest.json
{"report":"v2","rows":3}
$ curl -fsS --max-time 1 "$API/slow"
curl: (28) Operation timed out after N milliseconds with 0 bytes received
[exit status 28]
$ curl -fsS "http://127.0.0.1:$CLOSED_PORT/health"
curl: (7) Failed to connect to 127.0.0.1 port CLOSED_PORT after N ms: Couldn't connect to server
[exit status 7]
$ curl -fsS --retry 3 --retry-delay 1 "$API/busy"
curl: (22) The requested URL returned error: 503
curl: (22) The requested URL returned error: 503
{"status":"ok","version":"2.4.1"}

Without -f, curl printed the server’s error body and exited 0, so a script would carry on with an error message as its data. With -f it printed curl: (22) The requested URL returned error: 404 and exited 22; --fail-with-body, new in curl 7.76.0, kept the body for the log and still exited 22. -s hides the progress meter and the errors, and -S brings the errors back, which is why scripts write -fsS. When the code itself matters, -w '%{http_code}' prints it, here 404.

A redirect is not an error either. Without -L, curl saved the empty answer of the 302 into latest.json and exited 0; with -L it followed the Location and saved the report. Step down the stages of one call to see which status each failure leaves:

The last three lines of the session are the stalls. --max-time 1 ended the 3-second answer with status 28, and the closed port gave 7 at once; --connect-timeout would cap only the connection, and a name that no server can resolve gives 6. --retry 3 --retry-delay 1 met two 503 answers, printed both, and then got the 200: status 0. The curl 8.7.1 manual counts a timeout and the codes 408, 429, 500, 502, 503 and 504 as transient, waits 1 second before the first retry and doubles the wait after that unless --retry-delay fixes it, and stops retrying when --retry-max-time runs out. A policy of your own, with a deadline for the whole job, is a loop: see while, until and loop control. To turn a call you have tested by hand into code, the curl to code converter reads the same options.

A JSON answer needs its own check, and a token belongs in a file

Status 0 from curl -f says an answer arrived with a code below 400. It says nothing about what the answer says. The jq manual gives -e (--exit-status) the job: jq exits 0 when the last value it printed is neither false nor null, 1 when it is one of them, and 4 when it produced nothing at all. Put the two in a pipeline with pipefailpipefail: Bash's set -o pipefail option: the exit status of a pipeline is the rightmost non-zero stage's status, or 0 when every stage succeeded. A stage killed by SIGPIPE (because its reader closed early) is reported as 128 + 13 = 141. More in “set -e, set -u and pipefail: what they really do” set, as the lesson on errexit, nounset and pipefail explains, and the pipeline fails when either command does:

"""Checking a JSON answer with jq -e, and sending a token that sits in a mode-600 file instead of on the command line.

curl is macOS's /usr/bin/curl 8.7.1 and jq is 1.8.2. Both talk only to servers.web(), which this example starts on
127.0.0.1 and stops when it ends (API holds its address, shown as WEB_PORT). The token is a made-up placeholder that
only this little server knows.
"""
from session import env, mask, run, write
import servers

TOKEN = "EXAMPLE-NOT-A-SECRET"


def whoami(headers):
    if headers.get("Authorization") == f"Bearer {TOKEN}":
        return 200, '{"user":"deploy-bot","scopes":["read"]}\n'
    return 401, '{"error":"missing or wrong token"}\n'


web_port = servers.web({
    "/health": [(200, '{"status":"ok","version":"2.4.1","queue":3}\n')],
    "/health-b": [(200, '{"status":"degraded","version":"2.4.1","queue":4100}\n')],
    "/whoami": [(0, whoami)],
})
env("API", f"http://127.0.0.1:{web_port}")
mask(web_port, "WEB_PORT")
write("auth.header", f"Authorization: Bearer {TOKEN}\n")

run(r"""
set -o pipefail
curl -fsS "$API/health" | jq -e '.status == "ok"'
!curl -fsS "$API/health-b" | jq -e '.status == "ok"'
!curl -fsS "$API/health" | jq -e '.disk_free_gb'
version=$(curl -fsS "$API/health" | jq -er '.version') && echo "running $version"
# The header line lives in a file only its owner can read; curl reads the file itself.
chmod 600 auth.header
ls -l auth.header | cut -c1-10
curl -fsS -H @auth.header "$API/whoami"
!curl -fsS "$API/whoami"
""")

$ set -o pipefail
$ curl -fsS "$API/health" | jq -e '.status == "ok"'
true
$ curl -fsS "$API/health-b" | jq -e '.status == "ok"'
false
[exit status 1]
$ curl -fsS "$API/health" | jq -e '.disk_free_gb'
null
[exit status 1]
$ version=$(curl -fsS "$API/health" | jq -er '.version') && echo "running $version"
running 2.4.1
# The header line lives in a file only its owner can read; curl reads the file itself.
$ chmod 600 auth.header
$ ls -l auth.header | cut -c1-10
-rw-------
$ curl -fsS -H @auth.header "$API/whoami"
{"user":"deploy-bot","scopes":["read"]}
$ curl -fsS "$API/whoami"
curl: (22) The requested URL returned error: 401
[exit status 22]

.status == "ok" printed true for the healthy server and false, with status 1, for the degraded one, although curl succeeded both times. A field that is missing, .disk_free_gb, printed null and also failed, which is what a script wants from a typo or an API change. jq -er '.version' printed the raw value, so version=$(…) held 2.4.1, and the && ran only because the whole pipeline succeeded. More filters are in the lesson on jq, and the JSON formatter makes a long answer readable.

A token on curl’s command line is in its argument list, where other users of the same computer can read it, as the lesson on secure scripting shows. curl reads headers from a file instead: -H @auth.header sent the Authorization line from a file of mode 600, -rw-------, readable by its owner alone, and the server answered as deploy-bot; without the file it answered 401, and -f turned that into status 22. -K reads a whole set of options from a file in the same way. Keep the file in a secret storesecret store: Where a secret value lives at rest: a mode-600 file on the host, a dedicated store such as GitHub Actions secrets or HashiCorp Vault, or a password manager. A script reads the value from the store at each run and does not keep a copy of it. More in “Secure scripting: temp files, secrets and PATH” or write it with umask 077, and remove it when the job ends.

Bash can test a TCP port by itself through /dev/tcp

Before a script calls a service, it can ask whether anything is listening. The Bash manual lists /dev/tcp/host/port/dev/tcp redirection: A redirection to the name /dev/tcp/host/port, which Bash recognises and turns into a TCP connection to that host and port. No such file exists on the disk, and dash and POSIX sh do not have the feature. among the names Bash treats specially in a redirection: when the host and port are valid, Bash opens a TCP connection to them. Nothing on the disk has that name, and the session’s last line shows it: dash, which has no such feature, tried to create the file, and macOS has no /dev/tcp folder. Bash itself has the feature only when it was built with it, through the --enable-net-redirections option, on by default.

"""A TCP port check with nothing but Bash: the /dev/tcp/host/port redirection, wrapped in timeout.

Bash opens the connection itself (the Bash manual, Redirections); dash has no such name and macOS has no /dev/tcp
folder. The only ports involved are on 127.0.0.1: a small HTTP server this example starts and stops (WEB_PORT) and a
port nothing listens on (CLOSED_PORT). timeout is GNU coreutils 9.12.
"""
from session import env, mask, run
import servers

web_port = servers.web({"/health": [(200, '{"status":"ok"}\n')]})
closed_port = servers.closed()
env("WEB_PORT", web_port)
env("CLOSED_PORT", closed_port)
mask(web_port, "WEB_PORT")
mask(closed_port, "CLOSED_PORT")

run(r"""
port_open() { timeout 3 bash -c ': > "/dev/tcp/$1/$2"' port_open "$1" "$2"; }
port_open 127.0.0.1 "$WEB_PORT" && echo "WEB_PORT is open"
!port_open 127.0.0.1 "$CLOSED_PORT"
port_open 127.0.0.1 "$CLOSED_PORT" 2>/dev/null || echo "CLOSED_PORT: status $?"
# The same name opens a two-way descriptor: an HTTP request typed by hand.
exec 3<>"/dev/tcp/127.0.0.1/$WEB_PORT"
printf 'GET /health HTTP/1.0\r\n\r\n' >&3
head -n 1 <&3
exec 3<&-
!dash -c ': > /dev/tcp/127.0.0.1/80'
""")

$ port_open() { timeout 3 bash -c ': > "/dev/tcp/$1/$2"' port_open "$1" "$2"; }
$ port_open 127.0.0.1 "$WEB_PORT" && echo "WEB_PORT is open"
WEB_PORT is open
$ port_open 127.0.0.1 "$CLOSED_PORT"
port_open: connect: Connection refused
port_open: line 1: /dev/tcp/127.0.0.1/CLOSED_PORT: Connection refused
[exit status 1]
$ port_open 127.0.0.1 "$CLOSED_PORT" 2>/dev/null || echo "CLOSED_PORT: status $?"
CLOSED_PORT: status 1
# The same name opens a two-way descriptor: an HTTP request typed by hand.
$ exec 3<>"/dev/tcp/127.0.0.1/$WEB_PORT"
$ printf 'GET /health HTTP/1.0\r\n\r\n' >&3
$ head -n 1 <&3
HTTP/1.1 200 OK
$ exec 3<&-
$ dash -c ': > /dev/tcp/127.0.0.1/80'
dash: 1: cannot create /dev/tcp/127.0.0.1/80: Directory nonexistent
[exit status 2]

port_open passes the host and port to an inner bash -c as $1 and $2, never as text inside its script, and opens the connection with the null command :. The server’s port answered at once; the closed port printed two Connection refused lines and returned 1. The same name opens a two-way file descriptorfile descriptor: A small number a running program uses to name one of its open files or streams. A program normally starts with 0 (standard input), 1 (standard output) and 2 (standard error) already open. More in “Redirection: stdin, stdout, stderr, /dev/null”: exec 3<> connected descriptor 3, printf sent an HTTP request by hand, and head -n 1 read HTTP/1.1 200 OK, the status line whose number -w '%{http_code}' prints. The port number lookup says which service a port number usually belongs to.

The timeout 3 around it is for the case the session cannot show on one computer: a firewall that drops packets gives no refusal at all, so a connection attempt waits for the system’s own TCP timeout. GNU’s timeout manual says timeout then ends the command and exits with status 124, a number a script can test like any other; the lesson on signals, kill and timeout covers it in full.

Practise: what each remote command leaves behind

These questions change one detail of the sessions at a time: a value with a quote in it, a host built minutes ago, a far end without rsync, a redirect saved as an empty file, a 503 that clears on the third try, a field missing from the JSON. Work out the status and the output first; each explanation then names the recorded run behind its answer:

Interview questions

Warm-up (fresher to 2–5 years): after ssh host command returns, which side failed, and how can a script tell?

The exit status says which side failed. Per the EXIT STATUS section of ssh(1), ssh hands back the remote command’s own status, and keeps 255 for errors of its own. So 0 means the command ran and succeeded, 1 to 254 mean it ran and failed with that status, and 255 means ssh itself failed: no connection, a refused key, a changed host key, a timeout. A script therefore tests for 255 first and reports “could not reach the host” apart from “the command failed”. With BatchMode=yes and ConnectTimeout, that 255 comes within seconds instead of after a prompt that nobody answers. One caveat: a remote command that itself exits 255 cannot be told apart, so keep remote commands to ordinary statuses.

Three harder questions come next, each from a script that reached another machine and did the wrong thing there: a “fix” for a host-key error in a pipeline, a mirror that emptied a live folder, and a download that saved the wrong kind of answer. Sketch how you would begin each one, then open its first step:

Key takeaways

• ssh joins the words after the host into one string, and the far end’s login shell parses it again, so quote each value with printf %q or send the script on standard input with bash -s, whose arguments still need the quoting.
• A script’s ssh needs BatchMode=yes, a ConnectTimeout and server-alive messages so it fails in seconds; the first value in ssh_config wins, and status 255 means ssh failed, while any other status is the remote command’s.
• scp copies whole files and uses SFTP since OpenSSH 9.0, sftp -b runs checked batches and can resume, and rsync compares first, which needs rsync on both machines.
• rsync’s quick check skips files whose size and time match, which -c corrects at the cost of reading everything, and its delta sent 700 of 260,000 bytes for one changed byte; over ssh it reports 255, 23 or 24 when a copy fails.
• curl exits 0 on a 404 or a 302 unless -f and -L say otherwise, and 22, 28 and 7 name an HTTP error, a stall and a refused connection; jq -e with pipefail rejects a wrong answer, and -H @file keeps a token off the command line.
• /dev/tcp/host/port lets Bash open a TCP connection in a redirection, which makes a port check with timeout that needs no other tool, and which dash and POSIX sh do not have.