Error-handling patterns for scripts

A nightly job ran export.sh | gzip > daily.csv.gz and reported success for a week, yet each morning’s archive was 43 bytes long: export.sh had printed error: database locked on its standard output, where the data goes, and the pipeline’s status was the 0 of gzip. Nothing in the run said that anything had failed. A few error-handling patterns put the message on standard error and give the caller an exit status it can act on.

You will write errors to stderr while keeping stdout for data, branch on a command’s status with if, || and !, validate inputs and required commands before any side effect, choose exit codes a caller can act on, and keep batches going so the end of the run can list every failure.

Errors go to stderr; data stays on stdout

A shell program reads from standard input, writes data to standard output and writes diagnostics (errors, warnings, progress) to standard error. The split is not a convention of taste — it is the one that lets cmd | other-cmd work. The data stays on stdout, which other-cmd reads; the diagnostics still reach the terminal through stderr. POSIX defines standard error as the stream “for diagnostic output” (its stdin page), and the Google Shell Style Guide asks that every error message go there.

The one-line form is printf '…' >&2:

#!/bin/bash
# The one-line form for writing an error to standard error.
# Read-only snippet: run it from a bigger script.
printf 'error: no such directory: %s\n' "$dir" >&2
exit 3

>&2 sends the output to file descriptor 2, which is standard error unless the caller redirects it. Redirection: stdin, stdout and stderr explains redirection in full; here the one thing to carry is >&2 for every error print.

Small helpers for consistent errors: die, warn and a tool check

Two one-line helpers make every error look the same, and need_cmd fails early when a required outside tool is missing. errs.sh is the whole library:

from session import run, write

# A small library of error helpers. die prints to stderr and exits with a status, warn prints to stderr and returns,
# need_cmd checks for an outside command before any side effect runs. The script that uses them handles the three
# error paths: usage error (exit 2), missing input (exit 3), and record failure (exit 4), with messages a caller can
# read and act on.
write("errs.sh", """# Error helpers for the lesson's small library. source this file.
die()  { printf 'error: %s\\n' "$*" >&2; exit "${rc:-1}"; }
warn() { printf 'warning: %s\\n' "$*" >&2; }

need_cmd() {
    for c in "$@"; do
        if ! command -v -- "$c" >/dev/null 2>&1; then
            printf 'error: required command not found: %s\\n' "$c" >&2
            exit 1
        fi
    done
}
""")

write("import.sh", """#!/bin/bash
set -euo pipefail
# shellcheck source=errs.sh
source ./errs.sh

usage() {
    printf 'usage: import.sh <records-dir>\\n' >&2
    rc=2 die "wrong arguments"
}

[ $# -eq 1 ] || usage
dir=$1
[ -d "$dir" ] || { rc=3 die "no such directory: $dir"; }
need_cmd awk sort

failed=0
# Each file holds one record per line in the form "id,amount". A record whose amount is not above 0 counts as failed.
while IFS= read -r -d '' f; do
    while IFS=, read -r id amount; do
        if ! awk -v a="$amount" 'BEGIN { exit !(a > 0) }'; then
            warn "bad amount in $f: id=$id amount=$amount"
            failed=$((failed + 1))
            continue
        fi
        printf 'imported %s %s from %s\\n' "$id" "$amount" "$(basename -- "$f")"
    done < "$f"
done < <(find "$dir" -type f -name '*.csv' -print0 | sort -z)

if (( failed > 0 )); then
    rc=4 die "$failed record(s) failed; see warnings above"
fi
echo "every record imported"
""")

write("make_records.sh", """mkdir -p records
printf 'A,10\\nB,-5\\n' > records/jan.csv
printf 'C,20\\n' > records/feb.csv
""")

run("""
chmod +x import.sh make_records.sh
# usage error (no argument): exit 2 with a usage message to stderr
!bash import.sh
# missing input (bad path): exit 3
!bash import.sh ./no-such-dir
# normal run with a bad amount in one record: exit 4 with a count
bash make_records.sh
!bash import.sh ./records
# recover by writing the record cleanly and run again: exit 0
printf 'A,10\\nB,5\\n' > records/jan.csv
bash import.sh ./records
""")

$ chmod +x import.sh make_records.sh
# usage error (no argument): exit 2 with a usage message to stderr
$ bash import.sh
usage: import.sh <records-dir>
error: wrong arguments
[exit status 2]
# missing input (bad path): exit 3
$ bash import.sh ./no-such-dir
error: no such directory: ./no-such-dir
[exit status 3]
# normal run with a bad amount in one record: exit 4 with a count
$ bash make_records.sh
$ bash import.sh ./records
imported C 20 from feb.csv
imported A 10 from jan.csv
warning: bad amount in ./records/jan.csv: id=B amount=-5
error: 1 record(s) failed; see warnings above
[exit status 4]
# recover by writing the record cleanly and run again: exit 0
$ printf 'A,10\nB,5\n' > records/jan.csv
$ bash import.sh ./records
imported C 20 from feb.csv
imported A 10 from jan.csv
imported B 5 from jan.csv
every record imported

die writes error: … to stderr and exits with the status in $rc (default 1), so the caller writes rc=2 die "wrong arguments" on a usage error and rc=3 die "no such directory" on a missing input. warn writes warning: … to stderr and returns without exiting, so the caller can count the warnings and decide later. The session’s run showed four separate paths: the usage error with exit 2, the missing input with exit 3, the run with a bad record ending at exit 4 with the counts, and the clean run after the record was fixed.

need_cmd is the pattern for every outside tool the script relies on: command -v -- "$c" >/dev/null 2>&1 returns 0 when the shell can run the name c, whether it is a builtin, a function or a program on PATH. Checking the tools before the first side effect is the fail-early rule, which the next section unfolds.

Test a command’s status directly instead of saving it

A command’s exit statusexit status: The number from 0 to 255 every command ends with, kept by the shell in $?. 0 means success; Bash uses 126 for found but not executable and 127 for not found. More in “Your first script: shebang, chmod and running it” is a decision, so read it where the decision is made: as the condition of if (or while or until), on the left of || or &&, or after !. Those are also the places where set -e looks away, so a failure read there is handled instead of ending the script. Testing $? a few lines later is the slip ShellCheck’s SC2181 names, because every command in between sets $? to its own status.

from session import run, write

# Four patterns for reading an exit status directly, and the slip ShellCheck's SC2181 names: saving $? and testing it
# several lines later, when another command has already overwritten it.
write("status.sh", """#!/bin/bash
set -euo pipefail

# 1) Branch with "if cmd": the status is the condition; no "$? -eq 0" needed.
if grep -q error today.log; then
    echo "today has errors"
else
    echo "today is clean"
fi

# 2) Short-circuit with ||: handle the failure in-line and keep errexit on.
cd ./workspace || { echo "cannot enter workspace" >&2; exit 3; }
echo "in workspace: $PWD"
cd - >/dev/null

# 3) Negated condition with !: run a branch when the command fails.
if ! command -v nosuch_tool >/dev/null 2>&1; then
    echo "nosuch_tool is not installed; falling back to the shell builtin"
fi

# 4) Keep the number when it matters. Under set -e a plain "grep ...; rc=$?" would end the script when grep
#    returns 1, so the status is saved on the right of ||, where errexit looks away.
rc=0
grep -q panic today.log || rc=$?
case $rc in
    0)  echo "today has panic lines" ;;
    1)  echo "today has no panic lines (grep returned 1)" ;;
    *)  echo "grep itself failed with status $rc" >&2 ;;
esac
""")

write("sc2181.sh", """#!/bin/bash
# The SC2181 anti-pattern: save $? and test it several lines later. By then another command has reset $?.
# today.log holds "warn:" (lowercase). grep -q WARN returns 1 (no match), so the real answer is "no WARN".
grep -q WARN today.log
echo "about to test the status"
touch marker
if [ $? -ne 0 ]; then
    echo "no WARN found"
else
    echo "WARN found"
fi
""")

write("today.log", "fine\nwarn: cache expired\nerror: upstream 503\nfine\n")

run("""
chmod +x status.sh sc2181.sh
mkdir -p workspace
cat today.log
bash status.sh
echo "---the SC2181 anti-pattern:---"
bash sc2181.sh
!shellcheck sc2181.sh
""")

$ chmod +x status.sh sc2181.sh
$ mkdir -p workspace
$ cat today.log
fine
warn: cache expired
error: upstream 503
fine
$ bash status.sh
today has errors
in workspace: ~/demo/workspace
nosuch_tool is not installed; falling back to the shell builtin
today has no panic lines (grep returned 1)
$ echo "---the SC2181 anti-pattern:---"
---the SC2181 anti-pattern:---
$ bash sc2181.sh
about to test the status
WARN found
$ shellcheck sc2181.sh

In sc2181.sh line 7:
if [ $? -ne 0 ]; then
     ^-- SC2181 (style): Check exit code directly with e.g. 'if ! mycmd;', not indirectly with $?.

For more information:
  https://www.shellcheck.net/wiki/SC2181 -- Check exit code directly with e.g...
[exit status 1]

The four patterns the session shows are the whole kit:

• if cmd; then …; else …; fi — the condition is the command. No [ $? -eq 0 ]; it would read the same two values as cmd itself.
• cmd || { handler; exit 3; } — a short-circuit that handles the failure on the spot, with set -e still on. ShellCheck SC2164 reports a bare cd "$dir" and asks for the || exit form, which keeps a cd that failed from running the next line in the wrong folder.
• if ! cmd; then …; fi — the negation. ! inverts the status and lets a branch run on the failure.
• rc=0; cmd || rc=$?; case $rc in … — for the case where the number itself matters. Under set -e, a plain cmd; rc=$? never reaches the second command when cmd fails; on the right of || the status is saved instead, and the session’s grep -q panic ended with today has no panic lines (grep returned 1).

The anti-pattern in sc2181.sh was a short run with grep -q WARN today.log (no match, $? was 1), then an echo and a touch ($? reset to 0), then if [ $? -ne 0 ]; then echo "no WARN"; else echo "WARN found"; fi. The test read 0 -ne 0, which is false, so the script printed “WARN found” even though the log did not contain WARN. ShellCheck 0.11.0 reports this as SC2181.

Fail early: validate inputs before any side effect

A failure that stops a script after it has already written half the output is harder to clean up than a failure before any output. The order of the first lines of import.sh is the whole pattern — check the arguments, the input, and the required commands, then start work:

Three checks come before the first record is read. When the argument count is wrong, usage prints the one-line usage message and dies with 2. When the directory does not exist, die prints the path and dies with 3. When a required command is missing, need_cmd prints its name and exits with 1. In each case no record is processed and stdout stays empty, so the caller’s pipe reads nothing instead of half a result.

The habits the pattern names are small:

• [ $# -eq 1 ] || usage — the argument count is the first thing. Bash’s own builtins return 2 for incorrect usage (the Bash manual, Exit Status), and import.sh follows them with rc=2 die.
• [ -d "$dir" ] || die "no such directory: $dir" — the input is the second. -d checks for a directory; -e for any kind of file; -r for a readable file.
• need_cmd awk sort — every outside tool is checked once, at the top, so a run that misses a tool fails before any record is written.

Fail-early also applies inside the loop: when a record is bad, the loop collects it as a failure instead of writing a half-record. The next section covers that half of the pattern.

Choose exit codes on purpose, and summarise batches

Exit codes are the one thing a caller reads without any output. 0 means success and every other value means some kind of failure, which each program defines for itself: POSIX grep exits 0 when a line matched, 1 when none did and more than 1 on an error, and Bash’s builtins return 2 for incorrect usage. A script should choose its codes on a scale like that and document them, so a caller can branch on the number:

Statuses above 125 already mean something to a calling shell (the Bash manual: “the shell may use values above 125 specially”): 126 and 127 for a command that could not run, 128 + N for one killed by signal N. Keep a script’s own codes at 125 or below. And an exit code is a reason, not a count: the GNU Coding Standards warn that a status is 8 bits, so a count of 256 errors would arrive as 0 and read as success. import.sh picked 2, 3 and 4: a usage error (rc=2 die), a missing input (rc=3 die) and record failures (rc=4 die). A later wrapper could then retry on 4 (data problem) without retrying on 2 (bug in the call).

For a batch, keeping going is almost always better than stopping at the first failure. Record the id, the file and the reason in an array; at the end, print a summary and die with the right code:

#!/bin/bash
# A batch pattern: collect failures as the loop runs, then summarise and exit non-zero.
# Read-only snippet; die, warn and process are expected to come from the script's own helpers.
failed=()
for record in "${records[@]}"; do
    if ! process "$record"; then
        failed+=("$record")
        warn "failed: $record"
    fi
done

if (( ${#failed[@]} > 0 )); then
    printf 'summary: %d/%d records failed\n' "${#failed[@]}" "${#records[@]}" >&2
    printf '  %s\n' "${failed[@]}" >&2
    rc=4 die "see the summary above"
fi

The caller sees exit 4 and the summary on stderr; the stdout stream carries the records that did work. A rerun of the batch only needs the ids in the summary, not every record.

Practise: writing the right error path

The questions start from the two sessions and change one thing: where an error is written, a status read too late, a check that comes after the work, a code a caller must act on. Decide the outcome before you check:

Interview questions

Warm-up (fresher to 2–5 years): why do errors go to stderr instead of stdout, and how is >&2 read?

A program’s standard output is the data that the next command of a pipe, or a file, receives; standard error is a separate stream that stays on the terminal unless the caller redirects it. Writing errors to stderr keeps the pipeline working: a user who runs script | jq sees the data reach jq and the error reach the screen, instead of the error turning into unparseable input for jq. >&2 means “send this command’s output to file descriptor 2”, which is standard error (1 is standard output and 0 standard input; POSIX numbers them this way). So printf 'error: %s\n' "$msg" >&2 writes the formatted line to stderr, and a caller can keep the two streams apart with 2>errors.log.

Two questions follow: an import that reported success with records missing, and a build step that worked in the wrong folder although every status was 0. Note what you would look at first in each before opening it:

Key takeaways

• Errors go to stderr (>&2) so stdout stays a clean data stream for pipes and redirects; a one-line die and a one-line warn are enough for every script in this module.
• Read a command’s status where the decision is made: if cmd, cmd || …, ! cmd, and rc=0; cmd || rc=$? when the number matters. Testing $? after another command has run is ShellCheck’s SC2181 slip; the status is gone by then.
• Fail early: check the argument count, the input and every required command before the first side effect, so a broken call leaves no partial state.
• Choose exit codes a caller can act on: 2 for a usage error, as Bash’s builtins do, and 3 and up for failures of your own, kept at 125 or below, because a calling shell reads 126, 127 and 128 + N as “could not run” and “killed by a signal”.
• For a batch, keep going, collect the failures in an array, summarise them at the end, then die with the right code: the caller can retry only the failures without re-running the whole set.