Traps and cleanup: EXIT, ERR and signals

A batch job wrote its progress to ./workspace, crashed on a bad line of input, and left the folder behind. The next run opened the folder with yesterday’s half-finished files inside and treated them as its own, double-counting every bill. The missing line was one: a cleanup trap that runs however the script ends.

You will register code to run on exit with the trap builtin, learn the one way out that still skips it, catch SIGINT and SIGTERM so a worker finishes its current item and reports the right status, use ERR, RETURN and DEBUG for diagnostics, and single-quote the trap action so variables expand when the trap runs.

The trap builtin registers code to run on a signal

The traptrap: A Bash builtin that registers code to run on a signal or a shell event: EXIT when the shell exits, ERR when a command's failure would end a set -e script, RETURN when a function returns, DEBUG before every simple command. trap action signal …; the action is one shell word. builtin says “when this signal reaches the shell, run that code”. Its two arguments are the action, as one shell word, and one or more signal names (or EXIT, the shell’s own event). trap is one of POSIX’s special built-ins, so every POSIX shell has it, and the Bash manual describes it with the Bourne shell builtins.

The usual way to pair it with a temporary folder is mktemp -d and a cleanup action:

#!/bin/bash
# Pair mktemp -d with an EXIT trap so the folder is removed however the shell ends.
# Read-only snippet; a real script runs this at the top.
work=$(mktemp -d)
trap 'rm -rf -- "$work"' EXIT

The lesson’s examples use a fixed folder name so the recorded output is identical from run to run; a real script writes work=$(mktemp -d) and the trap removes it. tmp_cleanup_session.py writes three small scripts and runs them, then runs tmp_term.sh in Bash and in dash (the next section comes back to those two runs):

from session import run, write

# An EXIT trap paired with a temp folder is the usual way to make sure the folder is removed however the shell ends:
# a normal exit, a false under set -e, or a Ctrl-C from outside. The lesson's examples use a fixed name so the
# output is identical across runs; a real script writes "work=$(mktemp -d)" instead.
write("tmp_normal.sh", """#!/bin/bash
set -euo pipefail
work=./workspace
mkdir -p "$work"
trap 'rc=$?; rm -rf -- "$work"; echo "cleaned $work (rc=$rc)"' EXIT

touch -- "$work/report.json"
ls -- "$work"
echo "work done"
""")

write("tmp_errors.sh", """#!/bin/bash
set -euo pipefail
work=./workspace
mkdir -p "$work"
trap 'rc=$?; rm -rf -- "$work"; echo "cleaned $work (rc=$rc)"' EXIT

touch -- "$work/report.json"
echo "about to fail"
false
echo "unreachable"
""")

write("tmp_quoted_wrong.sh", """#!/bin/bash
# The classic double-quote slip: $work was empty when the trap was set, so the stored action is
# rm -rf -- ; echo "cleaned []" and rm gets no operand at all. Single-quote the action so the expansion happens
# when the trap runs, not when it is set.
work=
trap "rm -rf -- $work; echo \\"cleaned [$work]\\"" EXIT

work=./workspace
mkdir -p "$work"
touch -- "$work/leftover"
echo "after the script, the folder named $work still exists"
""")

# A TERM from outside, simulated by the script sending it to itself: the script sets no TERM trap at all.
write("tmp_term.sh", """#!/bin/bash
work=./workspace
mkdir -p "$work"
trap 'rm -rf -- "$work"; echo "cleaned $work"' EXIT
echo "working when TERM arrives"
kill -TERM $$
echo "unreachable"
""")

run("""
chmod +x tmp_normal.sh tmp_errors.sh tmp_quoted_wrong.sh tmp_term.sh
bash tmp_normal.sh
!bash tmp_errors.sh
bash tmp_quoted_wrong.sh
echo "leftover check: $(ls workspace/ 2>/dev/null)"
rm -rf ./workspace
!bash tmp_term.sh
echo "after bash: $(ls -d workspace 2>/dev/null || echo 'no workspace left')"
!dash tmp_term.sh
echo "after dash: $(ls -d workspace 2>/dev/null || echo 'no workspace left')"
rm -rf ./workspace
""")

$ chmod +x tmp_normal.sh tmp_errors.sh tmp_quoted_wrong.sh tmp_term.sh
$ bash tmp_normal.sh
report.json
work done
cleaned ./workspace (rc=0)
$ bash tmp_errors.sh
about to fail
cleaned ./workspace (rc=1)
[exit status 1]
$ bash tmp_quoted_wrong.sh
after the script, the folder named ./workspace still exists
cleaned []
$ echo "leftover check: $(ls workspace/ 2>/dev/null)"
leftover check: leftover
$ rm -rf ./workspace
$ bash tmp_term.sh
working when TERM arrives
cleaned ./workspace
[exit status 143]
$ echo "after bash: $(ls -d workspace 2>/dev/null || echo 'no workspace left')"
after bash: no workspace left
$ dash tmp_term.sh
working when TERM arrives
[exit status 143]
$ echo "after dash: $(ls -d workspace 2>/dev/null || echo 'no workspace left')"
after dash: workspace
$ rm -rf ./workspace

tmp_normal.sh finished its last line, the EXIT trap ran, printed cleaned ./workspace (rc=0), and the shell exited 0. tmp_errors.sh hit false under set -e, the EXIT trap still ran with rc=1 from the saved exit status, and the folder was gone before the shell left. The third one is a slip: with the action in double quotes, the $work inside expanded when the trap was set, while it was still empty, so the stored action was rm -rf -- ; echo "cleaned []", an rm with no operand at all. After the script the folder was still there, as the leftover check line showed. The section after next is about exactly that.

An EXIT trap runs however the script ends, except on SIGKILL

The EXIT pseudo-signal is the useful one. The Bash manual says the EXIT trap is executed when the shell exits, and in Bash 5.3 that means every way out but one. Step through five ways a script can end:

The one it misses is SIGKILL (kill -9 and the Linux Out-Of-Memory killer). signal(7) says it cannot be caught, blocked or ignored: the process ends at once, and no shell code runs. Everything else runs the trap in Bash: a plain exit, a false that set -e catches, a Ctrl-C that becomes SIGINT, and a kill -TERM from outside, which the session’s tmp_term.sh showed by sending itself TERM with no TERM trap set. That is Bash’s behaviour, not a rule of POSIX: dash 0.5.13.5 ran no EXIT trap for the same script and left the folder behind, so a script written for sh also traps INT and TERM and exits from there (trap 'exit 143' TERM).

The rc=$? on the first line of a cleanup action is a small habit that pays back. Without it, the next command in the trap resets $? to its own status; with it, the trap sees the status that caused the exit and can log it. The tmp_errors.sh run printed cleaned ./workspace (rc=1) only because the trap started with rc=$?.

Single-quote the trap action, so variables expand at trap time

The third script in the previous session showed the slip. The action is one shell word, and Bash reads it like any other string: with double quotes, every $name is expanded now, when the trap is being set; with single quotes the string reaches trap as it was written, and the expansion happens when the trap fires. ShellCheck’s SC2064 names the slip and the fix.

#!/bin/bash
# Trap action quoting: the string reaches `trap` once, and its expansions follow the quoting style.
# Read-only snippet.

# Wrong: $work expands now; the trap stores the current value.
trap "rm -rf -- $work" EXIT

# Right: $work expands when the trap runs, which is after the script has set $work.
trap 'rm -rf -- "$work"' EXIT

The same rule covers $LINENO, $BASH_COMMAND and $? inside an ERR or DEBUG trap: single-quote the action so each one reads the value that is current at trap time. A multi-line action is best written as a function and registered by name:

#!/bin/bash
# A multi-line cleanup lives in a function; the trap registers it by name.
# Read-only snippet; $work and $pidfile come from the surrounding script.
cleanup() {
    local rc=$?
    rm -rf -- "$work"
    [ -n "$pidfile" ] && rm -f -- "$pidfile"
    echo "exited with $rc" >&2
}
trap cleanup EXIT

A function also keeps the trap line short and puts the cleanup where it can be read, tested and changed like the rest of the script.

Catching INT and TERM, then exiting 130 or 143

SIGINT (number 2 on Linux and on macOS, written INT to trap) and SIGTERM (15, TERM) are the two polite “please stop” signals. Without a trap, Bash ends the script and reports 130 for INT (128 + 2) or 143 for TERM (128 + 15). With a trap, the handler runs — and it must call exit to leave; otherwise, after the handler returns, the shell goes back to what it was doing. A worker that finishes its current item and exits 143 is the standard form:

from session import run, write

# A worker that handles TERM, finishes the item it is on, cleans up and exits 143 (128 + 15, for SIGTERM).
# For a reproducible session, the script sends TERM to itself after the second item so no second shell is needed.
# In production, the TERM would come from a scheduler, a service manager stopping the job, or `kill` from an admin.
write("worker.sh", """#!/bin/bash
set -euo pipefail

STOP=0
cleanup() {
    local rc=$?
    rm -f -- state.json
    echo "cleaned state.json (rc=$rc)"
}
trap cleanup EXIT
trap 'STOP=1; echo "received TERM at item $i"' TERM

for i in 1 2 3 4; do
    echo "processing item $i"
    printf '{"last": %d}\\n' "$i" > state.json
    # Simulate an external 'kill -TERM' arriving while we are between items 2 and 3.
    if [ "$i" = 2 ]; then
        kill -TERM $$
    fi
    if (( STOP == 1 )); then
        echo "stopping gracefully after item $i"
        exit 143
    fi
done
echo "all items done"
""")

run("""
chmod +x worker.sh
!bash worker.sh
echo "worker exit status: $?"
""")

$ chmod +x worker.sh
$ bash worker.sh
processing item 1
processing item 2
received TERM at item 2
stopping gracefully after item 2
cleaned state.json (rc=143)
[exit status 143]
$ echo "worker exit status: $?"
worker exit status: 143

The handler set STOP=1 and printed the item at which TERM arrived. The main loop finished item 2, saw STOP, printed stopping gracefully after item 2, and exited 143. The EXIT trap then removed state.json and reported rc=143. The pattern:

• Set trap early, before the loop that handles items, so a signal between items is handled too.
• Keep the handler short. Set a flag and let the loop act on it. Bash runs a trap only between commands: while a foreground command such as sleep 60 runs, the handler waits for it to finish (the Bash manual, Signals). Start a long wait as sleep 60 & wait $! instead, because wait returns at once when a trapped signal arrives.
• Exit with the conventional status. exit 130 for INT and exit 143 for TERM give a caller that reads $? the same number as a death by that signal, while scripts that forget report 0 and look as though they finished the job. The script still ended normally, though, and a calling Bash script can tell: it treats a SIGINT as fatal only when the command it was waiting for was killed by it (the Bash manual, Signals). To end as if killed, reset the trap and send the signal again: trap - INT; kill -INT $$.

ERR, RETURN and DEBUG, and the subshell reset

Three Bash traps are not signals but shell events. The pseudo-signalA name that trap accepts in place of a signal for an event of the shell itself: EXIT when the shell exits, ERR when a command fails outside the spots where set -e looks away, RETURN when a function or sourced file returns, DEBUG before each command. names are:

• ERR: runs when a command fails, except in the spots where set -e looks away, whether or not set -e is on. Not inherited into a function unless set -E (errtrace) is on.
• RETURN: runs when a function or a sourced file returns to its caller. Not inherited unless set -T (functrace).
• DEBUG: runs before every simple command (and before a few compound ones, such as for and case). Also inherited only with set -T.

#!/bin/bash
# errtrace (set -E) makes the ERR trap inherit into functions and $() substitutions.
# Read-only snippet; a real script opens with this at the top.
set -eE
trap 'echo "ERR at line $LINENO: $BASH_COMMAND"' ERR

from session import run, write

# ERR runs when a command fails, outside the spots where set -e looks away. By default the ERR trap does not reach
# into a function; set -E (errtrace) makes it inherited. set -T (functrace) does the same for RETURN and DEBUG.
# A subshell resets every trap the shell has set, EXIT included (ignored signals stay ignored), so the parent's EXIT
# trap does not run when a subshell ends, and a trap set inside ( ... ) fires only when that subshell ends.
# trap -p lists what is set. POSIX.1-2024 added trap -p; dash 0.5.13.5 rejects it, and because trap is a special
# builtin the dash script ends with status 2.
write("err_demo.sh", """#!/bin/bash
set -eE
trap 'echo "ERR at line $LINENO: $BASH_COMMAND" >&2' ERR

echo "about to call a function"
greet() { echo "hello"; false; }
greet
echo "unreachable"
""")

write("subshell_reset.sh", """#!/bin/bash
trap 'echo "parent EXIT fired"' EXIT

( echo "inside a subshell that sets no trap of its own" )
(
    echo "inside subshell before setting its trap"
    trap 'echo "subshell EXIT fired"' EXIT
    echo "inside subshell after setting its trap"
)
echo "subshell ended; parent still running"
""")

run("""
chmod +x err_demo.sh subshell_reset.sh
!bash err_demo.sh
bash subshell_reset.sh
# trap -p lists what the shell has set:
bash -c "trap 'echo hi' EXIT; trap -p EXIT"
# POSIX.1-2024 added trap -p; dash 0.5.13.5 rejects it, and because trap is a special builtin the dash script ends:
!dash -c "trap 'echo hi' EXIT; trap -p EXIT; echo after"
""")

$ chmod +x err_demo.sh subshell_reset.sh
$ bash err_demo.sh
about to call a function
hello
ERR at line 6: false
[exit status 1]
$ bash subshell_reset.sh
inside a subshell that sets no trap of its own
inside subshell before setting its trap
inside subshell after setting its trap
subshell EXIT fired
subshell ended; parent still running
parent EXIT fired
# trap -p lists what the shell has set:
$ bash -c "trap 'echo hi' EXIT; trap -p EXIT"
trap -- 'echo hi' EXIT
hi
# POSIX.1-2024 added trap -p; dash 0.5.13.5 rejects it, and because trap is a special builtin the dash script ends:
$ dash -c "trap 'echo hi' EXIT; trap -p EXIT; echo after"
dash: 1: trap: Illegal option -p
hi
[exit status 2]

In the session, err_demo.sh has set -eE, so the ERR trap reached into greet and reported “ERR at line 6: false” the moment false failed. subshell_reset.sh showed the second rule, which the Bash manual states for every subshell: traps the shell has set are reset, EXIT included, while ignored signals (trap '' SIG) stay ignored. The first ( … ) sets no trap and ended without running the parent’s EXIT trap; the second set its own, which fired when it ended; the parent’s fired once, when the script ended.

trap -p lists what is currently set, printing a line per signal that reads like the trap command used to set it; POSIX.1-2024 added -p to the standard (its change history lists it for Issue 8), and Bash supports it. dash 0.5.13.5 does not yet, and because trap is a POSIX special builtin, a syntax error in it ends the shell: dash -c "trap -p EXIT; echo after" printed the usage error and then the EXIT handler’s output (“hi”), status 2, with no “after”. For a listing that works in every POSIX shell, dash included, run trap with no arguments.

Practise: predicting what a trap does

Each question changes one detail of the sessions: a signal name, a missing set -E, a double-quoted action, a long command running when TERM arrives. Predict the result first; the explanations name the run behind each answer:

Interview questions

Warm-up (fresher to 2–5 years): what does an EXIT trap, trap 'commands' EXIT, guarantee, and what does it not?

It guarantees that Bash runs commands when the shell exits: at the end of the script, at an exit, when set -e stops it, and, in Bash 5.3, when an INT, TERM or HUP with no trap of its own ends it. It does not cover SIGKILL (kill -9, the Linux OOM killer), which signal(7) says cannot be caught, and it does not promise that commands finish: a power cut can stop them half-way. It is also not portable: dash runs no EXIT trap when a signal ends the script, so a script for sh traps INT and TERM as well. Write the cleanup so that it is safe to run twice and touches only what the script created.

Two questions follow, both about cleanup that went wrong in production: a lock file that outlived its service, and a trap that removed a folder it was never meant to touch. Think through your first step for each before you open it:

Key takeaways

• trap 'action' SIGNAL registers action to run when SIGNAL (or EXIT, ERR, RETURN, DEBUG) reaches the shell. The action is one shell word; single-quote it so variables expand at trap time, as ShellCheck’s SC2064 names.
• In Bash, an EXIT trap runs however the shell ends (the last line, exit, a set -e stop, an INT or TERM) but never after a SIGKILL, which signal(7) says cannot be caught; dash runs it on none of the signals.
• A trapped INT or TERM runs the handler but does not exit on its own: write exit 130 for INT and exit 143 for TERM so the caller sees the right code, and keep the handler’s work small (set a flag; let the main loop finish its item, and wait for long commands with wait).
• ERR runs when a command fails outside the spots where set -e looks away; it is not inherited by functions unless set -E is on. RETURN and DEBUG need set -T for the same reason.
• A subshell resets every trap but ignored signals, so a trap set inside ( … ) fires only when the subshell ends; trap -p lists the current traps in Bash and in POSIX.1-2024, while dash 0.5.13.5 rejects -p, and because trap is a special builtin that error ends the dash script.