Background jobs, wait and parallel work

A script that checks 30 servers one after another spends almost all of its time waiting: 29 of every 30 seconds pass while the shell blocks on a probe it has already sent. Run the probes together and the same work ends in about a second, but only if the script still learns which servers answered and which did not.

You will start a job in the background with &, read its process id from $!, wait for each one and read its exit statusexit status: The number from 0 to 255 every command ends with, kept by the shell in $?. 0 means success; Bash uses 126 for found but not executable and 127 for not found. More in “Your first script: shebang, chmod and running it”, run a pool that keeps only a few jobs going at once, hand a list to xargs -P, and start a job that outlives the terminal with nohup.

A background job runs while the shell moves on

Ending a command with & starts it as a background jobbackground job: A command run with a trailing & so the shell starts it in a subshell and goes on without waiting; $! holds the PID of the most recent one, and wait collects its exit status. and lets the shell carry on with the next line. The Bash manual puts it plainly: a command ended with & runs asynchronously in a subshellsubshell: A copy of the shell that runs commands apart from it, as Bash does for each command of a pipeline. It starts with the shell's variables, and every change it makes to them is lost when it exits. More in “Reading files line by line with read”, the shell does not wait for it, and the return status is 0. So & tells you the job started, never how it ends. The special parameter $! holds the process id of the most recent background job, so read it the instant you start a job you mean to track:

#!/usr/bin/env bash
# Start two workers in the background and keep the PID of each one. $! holds the
# PID of the most recent background job, so read it right after the & that made it.
set -u
bash worker.sh alpha 0 &
first=$!
bash worker.sh beta 4 &
second=$!
echo "the shell started both workers and moved on"
# wait PID returns that job's own exit status, so each check is reported on its own
wait "$first"
echo "alpha exited $?"
wait "$second"
echo "beta exited $?"
cat report.alpha report.beta

The worker is a stand-in that writes its own report file and exits with a status. Each job writes a separate file, so two jobs never mix their lines on one stream. The session runs fanout.sh, then barewait.sh:

from session import files, run

# Background work is started inside script files run with `bash file.sh`: a
# non-interactive shell prints no "[1] 12345" job notice, so the output has no PID
# in it and is the same on every run.
files("worker.sh", "fanout.sh", "barewait.sh")
run("""
# two workers run at the same time; wait PID reports each one's own status
bash fanout.sh
# a bare wait waits for both, then returns 0 even though delta exited 5
bash barewait.sh
""")

# two workers run at the same time; wait PID reports each one's own status
$ bash fanout.sh
the shell started both workers and moved on
alpha exited 0
beta exited 4
report from alpha (rc=0)
report from beta (rc=4)
# a bare wait waits for both, then returns 0 even though delta exited 5
$ bash barewait.sh
bare wait returned 0

fanout.sh caught first=$! right after the first & and second=$! after the second, so it could later wait on each by name: alpha gave 0 and beta gave 4. barewait.sh instead ended with a bare wait, with no argument. A bare wait waits for every background job and then returns 0, whatever those jobs did, so it printed bare wait returned 0 even though one worker had exited 5. That gap is the whole reason the next section exists.

wait on a PID returns that job’s own status

To learn what a job did, wait on its PID. wait PID waits for that one job and returns its exit status, turning a background job back into a status you can test, just as a foreground command gives one. Naming several PIDs is a trap, though: wait "$a" "$b" waits for both but returns only the status of the last id listed, so the earlier ones are lost. The cure is to save each PID as its job starts and then wait on them one at a time, recording the failures. collect-status.sh does that for five checks:

#!/usr/bin/env bash
# Start every check at once, remember each job's PID in order, then wait for each
# PID and record the ones that failed. A background job's failure never stops the
# script, so the only way to know what failed is to collect each status yourself.
set -u
names=(web api db cache queue)
codes=(0 0 7 0 2)          # the status each check ends with; 0 is healthy
pids=()
for i in "${!names[@]}"; do
    bash check-one.sh "${names[$i]}" "${codes[$i]}" &
    pids+=("$!")
done
failed=()
for i in "${!pids[@]}"; do
    if ! wait "${pids[$i]}"; then
        failed+=("${names[$i]}")
    fi
done
if ((${#failed[@]})); then
    printf 'failed: %s\n' "${failed[*]}"
    exit 1
fi
echo "every check healthy"

Two checks are set to fail (db with 7, queue with 2). The loop over ${!pids[@]} waits on each saved PID and appends the name to failed when wait reports non-zero:

from session import files, run

files("collect-status.sh", "check-one.sh")
run("""
# five checks run at once; the script waits for each PID and names what failed
!bash collect-status.sh
# a failing background job does not stop a set -e script: the shell runs on
bash -c 'set -e; (exit 5) & wait; echo "still running after the failure, status $?"'
""")

# five checks run at once; the script waits for each PID and names what failed
$ bash collect-status.sh
failed: db queue
[exit status 1]
# a failing background job does not stop a set -e script: the shell runs on
$ bash -c 'set -e; (exit 5) & wait; echo "still running after the failure, status $?"'
still running after the failure, status 0

The script named failed: db queue and exited 1. The second line shows why you cannot lean on errexiterrexit: Bash's set -e option (set -o errexit): the shell ends the script when a simple command returns non-zero, outside the exemption spots (if/while/until condition, every command but the last of && or ||, after !, inside a function called from those, a command substitution, and non-last pipeline stages). More in “set -e, set -u and pipefail: what they really do” here: under set -e, a background job that exits 5 does not stop the script, which ran on and printed status 0. set -e watches the foreground command in front of it, and a job’s status arrives later, through wait. The statuses line up with the common signal convention too: a job stopped with kill PID (a SIGTERM) makes a later wait report 143, that is 128 + 15, as the exit status lesson sets out.

A job pool runs a few at a time

Starting every job at once with & is fine for a handful, but 200 jobs at once can exhaust connections, memory or file handles. A job pooljob pool: A way to run many jobs a few at a time: start up to a set number N, then wait for one to finish before starting the next, so the parallel work never exceeds the limit N. keeps the parallel work but caps it: run at most N jobs, and start the next only when one finishes. wait -n (added in Bash 4.3, the NEWS file records) is what makes a pool tidy: it returns as soon as any one job finishes, freeing a single slot, and wait -n -p pid (Bash 5.1) stores the PID of the job that just finished, so you can name it. Step through a pool of three slots running six checks:

A slot opened the moment app-2 finished, and the pool filled it at once rather than waiting for the whole batch; the one failure, app-3, went onto a list instead of slipping past. parallel-check.sh is that pool, and the session runs it, then shows the version gap and a portable fallback:

#!/usr/bin/env bash
# A job pool: run a list of checks with at most `limit` running at once. wait -n
# (Bash 4.3) returns as soon as any one job finishes, which frees a slot for the
# next, and -p (Bash 5.1) stores the finished job's PID so we can name its host.
set -u
limit=3
hosts=(app-1 app-2 app-3 app-4 app-5 app-6)
codes=(0 0 6 0 2 0)
declare -A host_of
failed=()
running=0
reap() {
    local pid status
    wait -n -p pid
    status=$?
    ((status == 0)) || failed+=("${host_of[$pid]}")
    ((running--))
}
for i in "${!hosts[@]}"; do
    bash check-one.sh "${hosts[$i]}" "${codes[$i]}" &
    host_of[$!]=${hosts[$i]}
    ((running++))
    ((running < limit)) || reap
done
while ((running > 0)); do
    reap
done
if ((${#failed[@]})); then
    mapfile -t failed < <(printf '%s\n' "${failed[@]}" | sort)
    printf 'failed: %s\n' "${failed[*]}"
    exit 1
fi
echo "every host ok"

from session import files, run

files("parallel-check.sh", "pool-portable.sh", "check-one.sh")
run("""
# six host checks, three at a time; the failed hosts are listed, sorted
!bash parallel-check.sh
# wait -n needs Bash 4.3; the Bash 3.2 that macOS ships rejects the option
!/bin/bash -c '(exit 1) & wait -n'
# the portable pool runs in batches, so it works in Bash 5.3 and in Bash 3.2 alike
!bash pool-portable.sh
!/bin/bash pool-portable.sh
""")

# six host checks, three at a time; the failed hosts are listed, sorted
$ bash parallel-check.sh
failed: app-3 app-5
[exit status 1]
# wait -n needs Bash 4.3; the Bash 3.2 that macOS ships rejects the option
$ /bin/bash -c '(exit 1) & wait -n'
/bin/bash: line 0: wait: -n: invalid option
wait: usage: wait [n]
[exit status 2]
# the portable pool runs in batches, so it works in Bash 5.3 and in Bash 3.2 alike
$ bash pool-portable.sh
failed: app-3 app-5
[exit status 1]
$ /bin/bash pool-portable.sh
failed: app-3 app-5
[exit status 1]

Both pools reported failed: app-3 app-5 and exited 1. But the middle line matters for anyone shipping a script: macOS still ships Bash 3.2, which has no wait -n and printed wait: -n: invalid option. pool-portable.sh avoids wait -n entirely: it runs the list in batches of three and waits for each whole batch with plain wait, so it ran the same on Bash 5.3 and on Bash 3.2. The trade-off is that one slow job in a batch holds up the next batch, which a wait -n pool avoids.

xargs runs commands in parallel for you

When the jobs are “run this one command over each of a list”, xargs -P is a pool in a single line, with no PIDs to track. -P N runs up to N copies at once and -n 1 hands each copy one item. It is a separate program, on macOS and Linux alike, so the shell’s version does not matter:

from session import files, run

files("grade.sh")
run("""
# xargs -P runs up to three copies at once; -n 1 hands each copy one argument
printf '%s\\n' alpha beta gamma delta | xargs -P 3 -n 1 bash grade.sh
# each job wrote its own file, so the lines are whole and in the order we read them
cat alpha.out beta.out gamma.out delta.out
# when any command ends non-zero, xargs ends non-zero too (this BSD xargs returns 1)
!printf '%s\\n' one two bad | xargs -P 2 -n 1 bash grade.sh
cat bad.out
""")

# xargs -P runs up to three copies at once; -n 1 hands each copy one argument
$ printf '%s\n' alpha beta gamma delta | xargs -P 3 -n 1 bash grade.sh
# each job wrote its own file, so the lines are whole and in the order we read them
$ cat alpha.out beta.out gamma.out delta.out
alpha: ok
beta: ok
gamma: ok
delta: ok
# when any command ends non-zero, xargs ends non-zero too (this BSD xargs returns 1)
$ printf '%s\n' one two bad | xargs -P 2 -n 1 bash grade.sh
[exit status 1]
$ cat bad.out
bad: failed

Each copy wrote its own .out file, so the lines stayed whole and the cat read them back in order; left to share one stream, parallel output interleaves. The last run shows the price of the convenience: when one command exited non-zero, xargs itself ended non-zero (this BSD xargs returned 1), but it gives one combined status for the whole run, not a per-job result. For which item failed, have each copy record its own result, as grade.sh did. Which tool fits depends on your case; answer three questions and let the table rule the others out:

In short: a loop of & with a per-PID wait when the list is small; a wait -n pool or xargs -P when you must cap how many run at once; a batch pool when the script must also run on Bash 3.2; and GNU Parallel, named here but not run because it is not installed, when you want grouped output and per-job codes and can install it.

nohup keeps a job running after logout

A background job still belongs to its shell. When the terminal closes, the shell receives SIGHUP and passes it on to its jobs, so a long job started with a plain & can die with the connection. nohup fixes that: it starts the command set to ignore SIGHUP, and the usual form redirects all three streams so the job never touches a terminal that may vanish.

#!/usr/bin/env bash
# Start a long job so that it survives the terminal closing. nohup makes the job
# ignore SIGHUP, and redirecting all three streams means the job never reads from
# or writes to a terminal that may vanish. The shell is then free to log out.
set -u
# > log sends standard output to a file, 2>&1 sends standard error the same way,
# and < /dev/null gives the job an input that is always at end of file.
nohup bash long-build.sh > build.log 2>&1 < /dev/null &
echo "build started; its PID is in \$! and its output is in build.log"
# disown would also remove the job from this shell's table so no SIGHUP is sent to
# it on exit; nohup already covers that, so the two are rarely both needed.

> build.log sends standard output to a file, 2>&1 sends standard error the same way, and < /dev/null gives the job an input that is always at end of file, so it never blocks waiting for a terminal. nohup(1) adds one rule worth knowing: if standard output is still a terminal, nohup appends the output to a file called nohup.out instead; with the explicit > build.log above, it does not. The other two tools people reach for do related jobs:

Practise: a parallel checker you can run

Write run_parallel N cmd_file, a script that reads one command per line from cmd_file and runs them with at most N going at once, exiting non-zero if any command failed. Build it from this lesson’s pieces: a wait -n -p pool that caps concurrency, a saved association from each PID to its command, and a failed list. Check it in your own shell against a small cmd_file of your own: a mix of true, false and sh -c 'exit 2' lines should make run_parallel 2 exit non-zero and name the failing lines, while a file of only true lines should exit 0. To prove the cap holds without timing anything, have each command append a line to a counter file on entry and remove it on exit, then check that the counter file never held more than N lines at once. A reference solution built this way exits 0 on the all-true file and non-zero, naming two failures, on the mixed file.

Interview questions

Warm-up (fresher to 2–5 years): what does $! hold, and what do wait PID and a bare wait each return?

$! holds the process id of the most recent background job, the one started by the last & (or a coprocess), so you read it straight after starting a job you want to track; the next background job overwrites it. wait PID waits for that one job and returns its own exit status, which is how you turn a background job back into a status you can test: a job that ran (exit 7) makes wait return 7, and a job killed by SIGTERM makes it return 143. A bare wait, with no argument, is different: it waits for every background job to finish and then returns 0, whatever those jobs did. So a bare wait is a barrier, not a status check; when a job may fail, save its PID and wait on it by name.

The two harder questions are about real breakages: a deploy that reported success while two hosts had failed, and a CI step that hung for its whole timeout though it returned at once on a laptop. Think about where you would look first in each case, then open the worked first step below:

Key takeaways

• & starts a background job and returns 0 at once, saying only that the job started; $! holds the PID of the most recent job, so read it immediately for a later wait.
• wait PID returns that job’s own exit status, while a bare wait returns 0, and naming several PIDs returns only the last, so collect failures one PID at a time; a background job’s failure never trips set -e.
• A job pool caps how many run at once: wait -n frees a slot as soon as any job finishes and wait -n -p names it, but wait -n needs Bash 4.3, so a script that must run on a stock macOS falls back to a batch of plain wait.
• xargs -P N is a one-line pool with a single combined status, and nohup cmd > log 2>&1 & keeps a job alive past a logout by ignoring SIGHUP and keeping it off the terminal.