Signals, kill and timeout

A script calls a probe that sometimes hangs for hours, and its caller needs the probe stopped after 20 seconds with a status that says “stopped for time”, not “failed”. Stopping a process means sending it a signal, and three questions decide what follows: which signal, which process receives it, and what that process does with it.

You will read the common signals and their numbers, send them with kill and test a process with kill -0, turn a status of 128 plus n back into a signal name, put a limit on a command with timeout, and write a launcher script that either hands TERM on to the program it starts or becomes that program with exec.

A signal is a numbered message with a default action

A signalsignal: A short message the kernel delivers to a process, with a name, a number and a default action such as ending it. A program can handle or ignore most signals; KILL and STOP cannot be caught. is a short message that the kernel delivers to a process. The keyboard produces some (Ctrl-C sends INT), the kernel produces others (SEGV for a bad memory access), and any process may send one to another with kill. Every signal has a name, a number and a default action: end the process, end it with a memory dump, ignore the signal, stop the process, or let a stopped one continue. A program may keep the default, ignore the signal, or install a handler that runs instead. In a shell the traptrap: A Bash builtin that registers code to run on a signal or a shell event: EXIT when the shell exits, ERR when a command's failure would end a set -e script, RETURN when a function returns, DEBUG before every simple command. trap action signal …; the action is one shell word. More in “Traps and cleanup: EXIT, ERR and signals” builtin installs the handler, as the traps lesson shows. Two signals are different: signal(7) says “The signals SIGKILL and SIGSTOP cannot be caught, blocked, or ignored.”

Look at the last column of the table first, because it separates the signals a handler can answer from the two it cannot. Six of the twelve change numbers between systems, so the table gives this Mac’s numbers (they come from kill -l, run below) beside Linux’s on x86-64 (from signal(7)).

HUP means the terminal went away, INT is Ctrl-C, TSTP is Ctrl-Z, TERM is the polite request that kill sends by default, and KILL is the forced stop. PIPE ends a writer whose reader has gone, which is the story behind status 141 in the pipelines lesson and the SIGPIPESIGPIPE: Signal 13, sent to a process that writes to a pipe nobody has open for reading any more, for example after head has exited. By default it ends the writer, and Bash reports that as status 128 + 13 = 141. More in “Pipelines, PIPESTATUS and pipefail” entry. Because six numbers move between systems, a script that runs on both writes names (kill -TERM, trap … TERM) and converts with kill -l when it must. This session lists the signals and converts in both directions:

from session import run

# The names and numbers this Mac's Bash uses. Linux numbers differ for some signals (signal(7)).
run("""
# the whole list, as number) name pairs
kill -l
# number to name, name to number
kill -l 15
kill -l TERM
# a status above 128 names the signal that killed the process
kill -l 143
kill -l 137
""")

# the whole list, as number) name pairs
$ kill -l
 1) SIGHUP	 2) SIGINT	 3) SIGQUIT	 4) SIGILL	 5) SIGTRAP
 6) SIGABRT	 7) SIGEMT	 8) SIGFPE	 9) SIGKILL	10) SIGBUS
11) SIGSEGV	12) SIGSYS	13) SIGPIPE	14) SIGALRM	15) SIGTERM
16) SIGURG	17) SIGSTOP	18) SIGTSTP	19) SIGCONT	20) SIGCHLD
21) SIGTTIN	22) SIGTTOU	23) SIGIO	24) SIGXCPU	25) SIGXFSZ
26) SIGVTALRM	27) SIGPROF	28) SIGWINCH	29) SIGINFO	30) SIGUSR1
31) SIGUSR2
# number to name, name to number
$ kill -l 15
TERM
$ kill -l TERM
15
# a status above 128 names the signal that killed the process
$ kill -l 143
TERM
$ kill -l 137
KILL

kill -l 15 printed TERM and kill -l TERM printed 15. The last two lines turned 143 and 137 into names: the Bash manual says kill -l takes a status above 128 to be that of a process ended by a signal, so it subtracts 128 for you. The third section builds on that.

kill sends TERM unless told otherwise, and a minus sign reaches a whole group

kill PID sends TERM. Another signal is named with -s NAME, -NAME, -NUMBER or -n NUMBER, and these spellings all mean the same thing. send.sh starts a sleeper for each spelling, signals it, and prints what wait returns. It also uses kill -0, which sends nothing: it only asks the kernel whether the process exists and whether you may signal it.

#!/bin/bash
# send.sh: send one signal to a sleeper in several spellings, and test a process with kill -0.

alive() {   # alive PID LABEL: kill -0 sends no signal; it only asks whether PID exists and may be signalled
    if kill -0 "$1" 2>/dev/null; then echo "$2: alive"; else echo "$2: gone"; fi
}

try() {     # try LABEL KILL-OPTIONS...: start a sleeper, signal it with those options, print what wait says
    local label=$1 pid
    shift
    sleep 47 &
    pid=$!
    kill "$@" "$pid"
    { wait "$pid"; } 2>/dev/null
    echo "$label -> wait says $?"
}

sleep 47 &
sleeper=$!
alive "$sleeper" "before any signal"
kill -0 "$sleeper"
echo "kill -0 returned $?"
kill "$sleeper"
{ wait "$sleeper"; } 2>/dev/null
echo "kill PID -> wait says $?"
alive "$sleeper" "after kill"

try 'kill -s TERM PID' -s TERM
try 'kill -TERM PID' -TERM
try 'kill -15 PID' -15
try 'kill -n 15 PID' -n 15
try 'kill -s KILL PID' -s KILL

from session import run, files

files("send.sh")

run("""
bash send.sh
""")

$ bash send.sh
before any signal: alive
kill -0 returned 0
kill PID -> wait says 143
after kill: gone
kill -s TERM PID -> wait says 143
kill -TERM PID -> wait says 143
kill -15 PID -> wait says 143
kill -n 15 PID -> wait says 143
kill -s KILL PID -> wait says 137

All four spellings of TERM gave 143, and -s KILL gave 137; the next section reads those numbers. The kill -0 lines show a test, not a promise: it returned 0 while the sleeper ran, and the helper reported gone after the kill. The process could also have exited a moment after the test, and the kernel may give its PID to an unrelated process later. A failed kill -0 does not prove that the process is gone either, since it also fails for a process that belongs to someone else (on this Mac, kill -0 1 printed Operation not permitted). Use it on processes you started.

kill PID reaches one process, but a job is often several. Every process belongs to a process groupA set of processes, usually one job, that share a group ID. The group's ID is the PID of its first process, and a signal sent to the group reaches every member.. A child starts in its parent’s group (credentials(7)), and a shell with job control gives each job a group of its own. A script runs with job control off, so its children share the script’s group; set -m turns it on, and then a background job leads a group named by its first PID. A minus sign in front of that number names the group, and -- keeps kill from reading -12345 as a signal option, as the POSIX kill page writes it. group.sh starts one job that starts two sleepers, kills only the job’s first process, and then kills the whole group:

#!/bin/bash
# group.sh: a job that starts two sleepers. Kill only the job's leader, then kill the whole group.
set -m                      # job control: every background job becomes a process group of its own

survivors() {               # how many of the recorded sleepers still exist
    local n=0 p
    while read -r p; do
        if kill -0 "$p" 2>/dev/null; then n=$((n + 1)); fi
    done < sleepers.txt
    echo "$n"
}

: > sleepers.txt
bash -c 'sleep 47 & echo $! >> sleepers.txt; sleep 47 & echo $! >> sleepers.txt; wait' &
leader=$!
until [ "$(wc -l < sleepers.txt)" -ge 2 ]; do sleep 0.05; done
echo "sleepers running: $(survivors)"

kill -TERM "$leader"
{ wait "$leader"; } 2>/dev/null
echo "leader killed, wait says $?, sleepers still running: $(survivors)"

kill -TERM -- "-$leader"    # a minus sign before the number names the process group
for _ in $(seq 100); do [ "$(survivors)" = 0 ] && break; sleep 0.05; done
echo "after kill -- -PGID, sleepers running: $(survivors)"

from session import run, files

files("group.sh")

run("""
# a job with two sleepers: kill the leader, then the whole group
bash group.sh
""")

# a job with two sleepers: kill the leader, then the whole group
$ bash group.sh
sleepers running: 2
leader killed, wait says 143, sleepers still running: 2
after kill -- -PGID, sleepers running: 0

The first kill ended the leader (143), and both sleepers went on running with no parent left to stop them. Then kill -TERM -- -$leader reached every member of the group and none remained. Without set -m there is no such group: a copy of group.sh without that line, run on this Mac, answered the group kill with No such process, because the job’s first process was not a group leader.

A process killed by signal n reports 128 plus n

In the last session wait returned 143 and 137 for sleepers that never called exit. The Bash manual gives the rule: when a fatal signal ends a command, Bash reports 128 plus that signal’s number as its status. The exit status lesson lists the conventional exit statusesexit status: The number from 0 to 255 every command ends with, kept by the shell in $?. 0 means success; Bash uses 126 for found but not executable and 127 for not found. More in “Your first script: shebang, chmod and running it”; this section reads the 128 range in both directions.

selfkill.sh lets a child shell send itself one signal. Given the word handler, it first installs one function for INT and TERM that prints caught and the name, then exits with 128 plus the signal’s number, computed with kill -l. deaths.sh runs the child once per signal and prints the status the parent sees:

#!/bin/bash
# selfkill.sh SIGNAL [handler]: send this shell one signal. With "handler", INT and TERM are caught by
# one function that reports the signal and exits with 128 plus its number.
on_signal() {
    echo "caught $1"
    exit $((128 + $(kill -l "$1")))
}
if [ "${2:-}" = handler ]; then
    trap 'on_signal INT' INT
    trap 'on_signal TERM' TERM
fi
kill -s "$1" $$
echo "survived $1"

#!/bin/bash
# deaths.sh [handler]: run selfkill.sh once per signal and print the status the parent sees.
for sig in HUP INT TERM KILL USR1; do
    { bash selfkill.sh "$sig" "$@"; } 2>/dev/null
    status=$?
    printf '%-4s status %3d = 128 + %-2d  kill -l says %s\n' \
        "$sig" "$status" "$((status - 128))" "$(kill -l "$status")"
done

from session import run, files

files("selfkill.sh", "deaths.sh")

run("""
# each child shell kills itself with one signal; the parent prints the status it sees
bash deaths.sh
# the same, with one handler for INT and TERM that exits with 128 plus the number
bash deaths.sh handler
""")

# each child shell kills itself with one signal; the parent prints the status it sees
$ bash deaths.sh
HUP  status 129 = 128 + 1   kill -l says HUP
INT  status 130 = 128 + 2   kill -l says INT
TERM status 143 = 128 + 15  kill -l says TERM
KILL status 137 = 128 + 9   kill -l says KILL
USR1 status 158 = 128 + 30  kill -l says USR1
# the same, with one handler for INT and TERM that exits with 128 plus the number
$ bash deaths.sh handler
HUP  status 129 = 128 + 1   kill -l says HUP
caught INT
INT  status 130 = 128 + 2   kill -l says INT
caught TERM
TERM status 143 = 128 + 15  kill -l says TERM
KILL status 137 = 128 + 9   kill -l says KILL
USR1 status 158 = 128 + 30  kill -l says USR1

In the first run HUP gave 129, INT 130, TERM 143 and KILL 137, each 128 plus the signal’s number, and kill -l turned every status back into its name. USR1 gave 158 because its number on this Mac is 30; the same death on Linux x86-64 reports 138 (128 + 10), another reason to translate with kill -l instead of remembering numbers. In the second run INT and TERM printed caught and still ended with 130 and 143. A program that answers a signal with a graceful shutdowngraceful shutdown: Stopping a program in an orderly way: it catches a request such as TERM, finishes or abandons its current work, releases what it holds and exits, instead of being killed on the spot. and exits 128 + n looks, to its caller, like a program the signal killed. HUP and KILL had no handler, so their lines did not change, and no handler can run for KILL.

timeout stops a runaway command and says how with 124 or 137

The probe from the opening needs a limit that nobody has to watch. timeout DURATION COMMAND runs the command and, when the duration is up, sends it TERM. It comes from GNU coreutils, and POSIX lists it too (“First released in Issue 8”, the 2024 edition). macOS has no timeout, so the sessions here run GNU coreutils 9.12’s. A duration is a number with an optional suffix of s, m, h or d (seconds by default), and 0 means no limit. Read the clock of timeout -k 5 30 cmd from the top in the figure and note which status each ending leads to:

The session runs the pieces with sleep 47 as the hanging command. A limit of 1 second marks a case where the time must run out, and 30 seconds a case where the command must finish in time:

from session import run, write

write("plain.txt", "")

run("""
# time runs out: timeout stops the command and returns 124
timeout 1 sleep 47; echo "status $?"
# the command finished in time: its own status comes through
timeout 30 sh -c 'exit 3'; echo "status $?"
# --preserve-status keeps the signal status of the command instead of 124
timeout --preserve-status 1 sleep 47; echo "status $?"
# -s picks the signal; KILL cannot be ignored, so timeout ends 137
timeout -s KILL 1 sleep 47; echo "status $?"
# a command that ignores TERM needs -k: after 1 more second, KILL goes out
timeout -v -k 1 1 bash -c 'trap "" TERM; sleep 47'; echo "status $?"
""")

run("""
# timeout failing by itself: 125 for its own error, 126 cannot run, 127 not found
timeout --nonsense 1 true; echo "status $?"
timeout 30 ./plain.txt; echo "status $?"
timeout 30 no-such-tool; echo "status $?"
""")

# time runs out: timeout stops the command and returns 124
$ timeout 1 sleep 47; echo "status $?"
status 124
# the command finished in time: its own status comes through
$ timeout 30 sh -c 'exit 3'; echo "status $?"
status 3
# --preserve-status keeps the signal status of the command instead of 124
$ timeout --preserve-status 1 sleep 47; echo "status $?"
status 143
# -s picks the signal; KILL cannot be ignored, so timeout ends 137
$ timeout -s KILL 1 sleep 47; echo "status $?"
status 137
# a command that ignores TERM needs -k: after 1 more second, KILL goes out
$ timeout -v -k 1 1 bash -c 'trap "" TERM; sleep 47'; echo "status $?"
timeout: sending signal TERM to command ‘bash’
timeout: sending signal KILL to command ‘bash’
status 137
# timeout failing by itself: 125 for its own error, 126 cannot run, 127 not found
$ timeout --nonsense 1 true; echo "status $?"
timeout: unrecognized option '--nonsense'
Try 'timeout --help' for more information.
status 125
$ timeout 30 ./plain.txt; echo "status $?"
timeout: failed to run command ‘./plain.txt’: Permission denied
status 126
$ timeout 30 no-such-tool; echo "status $?"
timeout: failed to run command ‘no-such-tool’: No such file or directory
status 127

The first line returned 124, which says only that the limit was reached. A command that finishes in time keeps its own status (exit 3 gave 3). --preserve-status swaps 124 for the status the command ended with: sleep died of TERM, so 143. -s KILL sends KILL at once and timeout reported 137. The -v -k 1 1 line shows why -k exists: the command ignored TERM with trap "" TERM, -v logged both signals, and one second after the first signal timeout sent KILL and ended with 137. The last three lines are failures of timeout itself or of the command it could not start, with the numbers a shell uses for the same causes:

-s SIGNAL changes the first signal, and -k DURATION adds KILL after that much more time. By default GNU timeout sends the signal to the command’s whole process group, so helpers the command started stop with it; --foreground is for a timeout that is not run from an interactive prompt, and the POSIX page describes -f as “Only time out the utility itself, not its descendants.”

A launcher must forward TERM to its child, or become the child with exec

Many scripts only start another program: a wrapper that sets the environment and runs the real one, or a service script that starts a worker. A signal sent to the launcher’s PID goes to the launcher, not to the program. The fix that keeps the launcher in charge is to run the program in the background, remember its PID and trap TERM to forward it. Step through one TERM sent to such a wrapper:

The wrapper waits twice for a reason. While Bash waits for a foreground command, it “will not execute the trap until the command completes”, so the wrapper waits on a background job instead. When a trapped signal arrives, the wait builtin “will return immediately with an exit status greater than 128”, and the trap runs right after. That first status is the signal’s, 128 + 15 for TERM, and the child is still shutting down. A second wait collects the real status. app.sh is the program; it saves its state on TERM and exits 7:

#!/bin/bash
# app.sh: the program being supervised. On TERM it saves its state and exits 7.
trap 'echo "TERM received, saving state" >> app.log; exit 7' TERM
echo "ready" >> app.log
while :; do sleep 0.1; done

#!/bin/bash
# wrap.sh: run app.sh as a child, hand TERM on to it, and leave with the child's own status.
trap 'echo "forwarding TERM" >> wrap.log; kill -TERM "$child"' TERM
bash app.sh &
child=$!
echo "started the child" >> wrap.log
wait "$child"
status=$?
echo "first wait returned $status" >> wrap.log
wait "$child"
status=$?
echo "second wait returned $status" >> wrap.log
exit "$status"

nowrap.sh is wrap.sh without the trap line, and drive.sh starts a wrapper, waits until the wrapper and app.sh are both up, sends TERM to the wrapper only, and prints both logs. The session runs it on both wrappers:

from session import run, files

files("app.sh", "wrap.sh", "nowrap.sh", "drive.sh")

run("""
# TERM goes to the wrapper only; the wrapper hands it on and waits twice
bash drive.sh wrap.sh
# the same wrapper without the trap
bash drive.sh nowrap.sh
""")

# TERM goes to the wrapper only; the wrapper hands it on and waits twice
$ bash drive.sh wrap.sh
wrap.sh exited with 7
--- wrap.log
started the child
forwarding TERM
first wait returned 143
second wait returned 7
--- app.log
ready
TERM received, saving state
# the same wrapper without the trap
$ bash drive.sh nowrap.sh
nowrap.sh exited with 143
--- wrap.log
started the child
--- app.log
ready
app still running after its wrapper died: yes

The first run is the figure on real logs: first wait returned 143, then second wait returned 7, the app’s own line about TERM, and a wrapper that exited 7. The two logs are separate files on purpose, because the wrapper’s first line and the app’s handler run at nearly the same moment and a shared file would order them differently from run to run. Without the trap, TERM ended the wrapper with 143 and app.sh kept running, never signalled; drive.sh found it with kill -0 and ended it.

The other fix is to make the launcher disappear. exec COMMAND replaces the shell with the command: the same process, the same PID, and no parent left to take the signal. (The file descriptors lesson covers the form of exec that only changes redirections.) In the next session start-plain.sh runs the service as a child, start-exec.sh ends with exec, and launch.sh sends TERM to the PID it started and reports whether launcher and service shared it:

from session import run, files

files("serve.sh", "start-plain.sh", "start-exec.sh", "launch.sh")

run("""
# TERM goes to the PID that was started, in both runs
bash launch.sh start-plain.sh
bash launch.sh start-exec.sh
""")

# TERM goes to the PID that was started, in both runs
$ bash launch.sh start-plain.sh
launcher and service share one PID: no
start-plain.sh exited with 143
serve.sh: ready
service still running: yes
$ bash launch.sh start-exec.sh
launcher and service share one PID: yes
start-exec.sh exited with 0
serve.sh: ready
serve.sh: TERM received, shutting down
service still running: no

With exec, TERM went straight to the service’s handler, which logged its line, and wait returned the service’s own 0. Without it the launcher died of TERM (143) and the service stayed up, unaware. Use exec when the launcher has nothing left to do after the program ends, and a forwarding wrapper when it must still act or report afterwards.

Practise: stop a worker and cap a hang

Write worker.sh, a loop that appends tick 1, tick 2 and so on to worker.log, one every 0.2 seconds, traps TERM, appends bye and exits 143. Write limit.sh, which runs timeout 1 sleep 30 and prints stopped by timeout when the status is 124 and finished otherwise. Check both in your own shell without a clock. Start the worker in the background, wait until worker.log holds a tick 1 line, send TERM with kill, and wait for it: wait must return 143, the last line of the log must be bye, and no tick line may appear after it. Then bash limit.sh must print stopped by timeout once, and with true in place of sleep 30 it must print finished.

Interview questions

Three jobs ended with statuses 130, 137 and 143. What does each number tell you, and what produced it?

Each is 128 plus a signal number, the status Bash reports for a command that a signal killed. 130 is SIGINT (2): Ctrl-C or kill -INT. 143 is SIGTERM (15): the polite stop that kill, a service manager or a scheduler sends, which a program can catch to clean up. 137 is SIGKILL (9): it cannot be caught, so the program got no chance to clean up, and typical senders are kill -9, the kernel’s out-of-memory killer and a timeout whose -k period ran out. Two cautions apply. A program can also choose to exit 130 or 143 after handling a signal, so the number alone does not prove that one arrived. And the numbers follow the system’s signal numbering, so kill -l 137 is the safe way to translate a status.

Two harder questions are about a log line that disagrees with what is still running, and about a stop that takes longer than it should and says nothing while it does. Sketch the process tree for each case on paper before you open its first step.

Key takeaways

• A signal is a numbered message with a default action: TERM can be caught, ignored or answered with cleanup, KILL and STOP cannot, and six common numbers differ between macOS and Linux, so scripts write names.
• kill sends TERM unless told otherwise, kill -0 only tests that a process exists and may be signalled, and kill -- -PGID reaches every process of a group, which a script has per job only with set -m.
• A command killed by signal n reports 128 + n (130 for INT, 137 for KILL, 143 for TERM), kill -l STATUS names the signal, and a handler that exits 128 + n gives the same number as a kill.
• timeout returns 124 when the limit was reached, 137 when KILL was needed (-k, -s KILL), the command’s own status otherwise, and 125, 126 or 127 for its own failures; --preserve-status keeps the command’s status.
• A launcher either traps TERM, forwards it to its child and waits twice (the first wait returns 128 + the signal at once, the second returns the child’s status), or replaces itself with exec so that the program gets the signal.