while, until, break, continue and select

A retry loop written as for attempt in 1 2 3 4 5; do upload && break; sleep $(( 2 ** attempt )); done pauses for 62 seconds when every upload fails, and then the script carries on with status 0 as if one had worked. The same five attempts need only 30 seconds of pauses, and they can end with the real failure, once the loop knows when to stop.

You will write loops that run while a command succeeds or until it does, leave or skip the passes of nested loops with break and continue, build that retry loop and a polling loop with a deadline, and offer a numbered menu with select.

while runs while a command succeeds, until while it fails

A retry policy that starts at 2 seconds and doubles pauses for 2, 4, 8 and 16 seconds if it stops before 32. A while loop can print that schedule. It runs its test, which is any command, and while the test ends with exit statusexit status: The number from 0 to 255 every command ends with, kept by the shell in $?. 0 means success; Bash uses 126 for found but not executable and 127 for not found. More in “Your first script: shebang, chmod and running it” 0 it runs the body and tests again. until is the same loop with the test turned round: it runs the body while the test fails and stops at the first success. The session prints the schedule with while, with until and once more in dash:

from session import run

run("""
delay=2; while (( delay <= 16 )); do echo "pause ${delay}s"; delay=$(( delay * 2 )); done; echo "after the loop: delay=$delay"
delay=2; until (( delay > 16 )); do echo "pause ${delay}s"; delay=$(( delay * 2 )); done
dash -c 'delay=2; while [ "$delay" -le 16 ]; do echo "pause ${delay}s"; delay=$((delay * 2)); done'
touch report.1 report.2 report.3
n=1; while [[ -e report.$n ]]; do n=$(( n + 1 )); done; echo "next free name: report.$n"
false; while false; do echo "never printed"; done; echo "status after no pass: $?"
check() { return 3; }; tries=0; until check; do echo "check failed with $?"; tries=$(( tries + 1 )); (( tries == 2 )) && break; done
""")

$ delay=2; while (( delay <= 16 )); do echo "pause ${delay}s"; delay=$(( delay * 2 )); done; echo "after the loop: delay=$delay"
pause 2s
pause 4s
pause 8s
pause 16s
after the loop: delay=32
$ delay=2; until (( delay > 16 )); do echo "pause ${delay}s"; delay=$(( delay * 2 )); done
pause 2s
pause 4s
pause 8s
pause 16s
$ dash -c 'delay=2; while [ "$delay" -le 16 ]; do echo "pause ${delay}s"; delay=$((delay * 2)); done'
pause 2s
pause 4s
pause 8s
pause 16s
$ touch report.1 report.2 report.3
$ n=1; while [[ -e report.$n ]]; do n=$(( n + 1 )); done; echo "next free name: report.$n"
next free name: report.4
$ false; while false; do echo "never printed"; done; echo "status after no pass: $?"
status after no pass: 0
$ check() { return 3; }; tries=0; until check; do echo "check failed with $?"; tries=$(( tries + 1 )); (( tries == 2 )) && break; done
check failed with 3
check failed with 3

The table follows the first loop one test at a time. The body doubles delay, so the test eventually fails and the loop ends:

until (( delay > 16 )) printed the same four lines, because “until the delay passes 16” is “while it does not”. The dash line is the POSIX form of the same counter: (( )) is a Bash command that POSIX does not have, as the shell arithmetic lesson showed, so a script for sh tests with [ "$delay" -le 16 ] and computes with $((delay * 2)). A while loop with a counter is also how sh counts when the C-style for of the for loops lesson is not there.

The test does not have to be arithmetic. while [[ -e report.$n ]] went on adding 1 while a file of that name existed, and stopped at report.4, the first name nobody had taken.

Two rules about statuses come from the manual and POSIX. A loop ends with the status of the last command its body ran, or 0 when the body never ran: while false ran no pass, so the session printed 0 although false had failed. The test’s own status is not the loop’s status. Inside the body of until, though, $? still holds the status of the failed test, which is how until check could print “check failed with 3”, and how a loop can keep it for later.

break leaves a loop, continue skips a pass, and a number reaches outer loops

: is a builtin that always succeeds, so while : never stops by itself; break ends it. The session’s first line counted attempts 1, 2 and 3 and left after the third, the shape to use when the test belongs at the end of the body, because the shell has no do-while loop. continue does the opposite of finishing: it skips the rest of the body and starts the next pass, after testing again in while and until, or with the next word in for.

from session import files, run, write

files("search.sh")
write("mon/a.log", "INFO backup started\nINFO backup done\n")
write("mon/b.log", "INFO 12 files copied\n")
write("tue/a.log", "ARCHIVED to cold storage\n")
write("tue/b.log", "ERROR this old line is never read\n")
write("wed/a.log", "INFO backup started\nERROR disk full\n")
write("wed/b.log", "ERROR not reached either\n")
write("thu/a.log", "INFO backup done\n")
run("""
attempt=0; while :; do attempt=$(( attempt + 1 )); echo "attempt $attempt"; (( attempt == 3 )) && break; done; echo "left the loop after $attempt"
for i in 1 2; do for c in a b c; do [ "$c" = b ] && continue; [ "$i" = 2 ] && break 2; echo "$i$c"; done; done; echo "after both loops"
dash -c 'for i in 1 2; do for c in a b c; do [ "$c" = b ] && continue; [ "$i" = 2 ] && break 2; echo "$i$c"; done; done; echo "after both loops"'
bash search.sh
""")

$ attempt=0; while :; do attempt=$(( attempt + 1 )); echo "attempt $attempt"; (( attempt == 3 )) && break; done; echo "left the loop after $attempt"
attempt 1
attempt 2
attempt 3
left the loop after 3
$ for i in 1 2; do for c in a b c; do [ "$c" = b ] && continue; [ "$i" = 2 ] && break 2; echo "$i$c"; done; done; echo "after both loops"
1a
1c
after both loops
$ dash -c 'for i in 1 2; do for c in a b c; do [ "$c" = b ] && continue; [ "$i" = 2 ] && break 2; echo "$i$c"; done; done; echo "after both loops"'
1a
1c
after both loops
$ bash search.sh
checked mon/a.log
checked mon/b.log
skipped tue: archived
first error in wed/a.log
search finished

In nested loops a number says which loop is meant, counted from the inside. In the second line, continue skipped the echo when c was b, and break 2 ended both loops as soon as i was 2, so only 1a and 1c were printed, in Bash and in dash alike. POSIX adds that a number larger than the loops around the command leaves the outermost one. The four forms, as they act from inside an inner loop:

search.sh uses both numbers on four folders of logs: a folder whose log starts with ARCHIVED is skipped as a whole, and the first ERROR line anywhere ends the search.

#!/usr/bin/env bash
# search.sh: finds the first log with an ERROR line, folder by folder.
# A folder whose log starts with ARCHIVED is skipped as a whole.
for day in mon tue wed thu; do
  for log in "$day"/*.log; do
    if grep -q '^ARCHIVED' "$log"; then
      echo "skipped $day: archived"
      continue 2
    fi
    if grep -q ERROR "$log"; then
      echo "first error in $log"
      break 2
    fi
    echo "checked $log"
  done
done
echo "search finished"

Step through the run to see which files the two numbers kept the script from opening:

tue/b.log holds an ERROR line, yet the search reported wed/a.log: continue 2 had already moved on to the next day, as intended for an archived folder. A plain continue there would have opened tue/b.log and stopped the search on an old error.

A retry loop needs a limit, a growing pause and a check after the last try

The opening’s loop shows what goes wrong. The session replaces sleep with a function that prints the pause instead of waiting, and lets upload fail every time:

from session import files, run

files("retry.sh", "flaky.sh")
run("""
# upload stands in for a command that always fails; this sleep prints the pause instead of waiting
upload() { return 1; }; sleep() { echo "(sleep $1)"; }
for attempt in 1 2 3 4 5; do upload && break; sleep $(( 2 ** attempt )); done; echo "carried on, status $?"
unset -f sleep
RETRY_SLEEP=: bash retry.sh bash flaky.sh
!RETRY_SLEEP=: bash retry.sh false
""")

# upload stands in for a command that always fails; this sleep prints the pause instead of waiting
$ upload() { return 1; }; sleep() { echo "(sleep $1)"; }
$ for attempt in 1 2 3 4 5; do upload && break; sleep $(( 2 ** attempt )); done; echo "carried on, status $?"
(sleep 2)
(sleep 4)
(sleep 8)
(sleep 16)
(sleep 32)
carried on, status 0
$ unset -f sleep
$ RETRY_SLEEP=: bash retry.sh bash flaky.sh
flaky: call 1
retry: attempt 1 failed with status 3, next in 2s
flaky: call 2
retry: attempt 2 failed with status 3, next in 4s
flaky: call 3
retry: succeeded on attempt 3
$ RETRY_SLEEP=: bash retry.sh false
retry: attempt 1 failed with status 1, next in 2s
retry: attempt 2 failed with status 1, next in 4s
retry: attempt 3 failed with status 1, next in 8s
retry: attempt 4 failed with status 1, next in 16s
retry: giving up after 5 attempts, last status 1
[exit status 1]

The naive loop printed five pauses, 2 + 4 + 8 + 16 + 32 = 62 seconds, the last one after the final failure, when no attempt was left to wait for. Then it reported status 0, because a for loop ends with the status of its last command, and that was sleep. A script that goes on to delete the local copy would delete it after five failed uploads.

retry.sh fixes all three problems: the wasted last pause, the lost status and the script that carries on. It runs the command as the test of until, so the body runs only after a failure. The body saves the status, gives up with that status after attempt 5, and otherwise pauses 2 ** attempt seconds, an exponential back-offexponential back-off: A retry policy whose pause grows by a fixed factor after every failed attempt, such as 2, 4, 8 and 16 seconds, so that a struggling service gets more time to recover with each try. that gives a struggling service more time with every try:

#!/usr/bin/env bash
# retry.sh COMMAND [ARGUMENT...]: runs COMMAND up to 5 times, pausing 2, 4, 8 and 16 seconds
# between attempts, and exits with the status of the last attempt.
# RETRY_SLEEP=: skips the pauses (the session sets it, so the page shows them without waiting).
max=5
attempt=1
until "$@"; do
  status=$?
  if (( attempt == max )); then
    echo "retry: giving up after $attempt attempts, last status $status" >&2
    exit "$status"
  fi
  pause=$(( 2 ** attempt ))
  echo "retry: attempt $attempt failed with status $status, next in ${pause}s" >&2
  "${RETRY_SLEEP:-sleep}" "$pause"
  attempt=$(( attempt + 1 ))
done
echo "retry: succeeded on attempt $attempt"

The session set RETRY_SLEEP=:, so the pauses were printed but not waited. flaky.sh failed twice with status 3 and succeeded on its third call, after 2 + 4 = 6 seconds of pauses. With false, every attempt failed: the pauses came to 2 + 4 + 8 + 16 = 30 seconds, and the script exited with 1, the status of the last attempt. Step through the calls:

Pollingpolling: Checking a state again and again at a fixed interval, such as testing once a second whether a file exists, until it holds or a deadline passes. is the same loop pointed at a state instead of a command’s success. wait-for-file.sh tests once a second whether a file exists and gives up at a deadline taken from SECONDS, which Bash keeps as the number of seconds since the shell started, read from the system clock at one-second resolution. So a limit of 2 is good to about a second. It sleeps whole seconds because POSIX’s sleep promises nothing finer.

#!/usr/bin/env bash
# wait-for-file.sh FILE LIMIT: checks once a second until FILE exists,
# and gives up when LIMIT seconds have passed.
file=$1
limit=$2
deadline=$(( SECONDS + limit ))
until [[ -e $file ]]; do
  if (( SECONDS >= deadline )); then
    echo "wait-for-file: gave up, $file did not appear" >&2
    exit 1
  fi
  sleep 1
done
echo "wait-for-file: $file is there"

In the session it gave up on a file nobody created, with status 1, and found ready.flag once a background job created it two seconds later:

from session import files, run

files("wait-for-file.sh")
run("""
!bash wait-for-file.sh missing.flag 2
(sleep 2; touch ready.flag) & bash wait-for-file.sh ready.flag 10
""")

$ bash wait-for-file.sh missing.flag 2
wait-for-file: gave up, missing.flag did not appear
[exit status 1]
$ (sleep 2; touch ready.flag) & bash wait-for-file.sh ready.flag 10
wait-for-file: ready.flag is there

select prints a numbered menu and puts the answer in REPLY

select turns a list of words into a menu. It prints each word with a number on standard error, prompts with PS3, and reads one line from standard input. When the line is the number of a word, the loop variable takes that word, any other text empties it, and the line itself always lands in REPLY. The body runs after every line that is not empty, and the loop prompts again until a break ends it. Here is a menu of three service actions:

#!/usr/bin/env bash
# menu.sh: a numbered menu of service actions; the choice comes from standard input.
PS3='action number: '
select action in start stop status; do
  if [[ -n $action ]]; then
    echo "chose $action (REPLY=$REPLY)"
    break
  fi
  echo "no action has the number $REPLY"
done

The session feeds the answers through a pipe, so the typed numbers do not appear after the prompt:

from session import files, run

files("menu.sh")
run("""
printf '2\\n' | bash menu.sh
printf '9\\n\\n3\\n' | bash menu.sh
printf '1\\n' | bash menu.sh 2> /dev/null
!bash menu.sh < /dev/null
!dash menu.sh
""")

$ printf '2\n' | bash menu.sh
1) start
2) stop
3) status
action number: chose stop (REPLY=2)
$ printf '9\n\n3\n' | bash menu.sh
1) start
2) stop
3) status
action number: no action has the number 9
action number: 1) start
2) stop
3) status
action number: chose status (REPLY=3)
$ printf '1\n' | bash menu.sh 2> /dev/null
chose start (REPLY=1)
$ bash menu.sh < /dev/null
1) start
2) stop
3) status
action number:
[exit status 1]
$ dash menu.sh
menu.sh: 4: Syntax error: "do" unexpected
[exit status 2]

Each input line had its own effect, as the table shows:

The menu came back only after the empty line: since bash-2.05b, select reprints it only when REPLY is empty, as its NEWS file records. 2> /dev/null hid the menu and the prompt and left “chose start”, which shows they go to standard error, so a script’s output stays clean when its menu is used. dash stopped at do with a syntax error and status 2: POSIX lists select among the words a shell may reserve with unspecified results, and dash has no such loop.

Practise: loops that stop on time

These questions move the lesson’s loops onto new cases: other file names, a step that jumps over its target, nested loops with continue 2, a retry whose pauses grow as 3 ** attempt, and menu.sh at end of input. Work out what each prints, or how long it waits, before you check your answer:

Interview questions

Warm-up (fresher to 2–5 years): how do while and until differ, and what is while : for?

Both run a test command before every pass. while runs the body as long as the test succeeds, with status 0, and until runs it as long as the test fails, so until cmd reads as “while not cmd”. The colon is a builtin that always succeeds, so while : loops until a break inside the body ends it: that is how the shell writes a loop whose exit test sits at the end, since it has no do-while. Either loop ends with the status of the last body command it ran, or 0 when the body never ran, so a loop’s status does not say whether its condition ever held.

Two harder questions follow: one about deciding which failures a retry wrapper should retry and how long it may take in all, and one about a search whose break stopped working after a refactor. Decide what you would check first, then open each first step:

Key takeaways

• while runs its body while a test command succeeds and until while it fails; the test can be any command, and the loop ends with the status of the last body command, or 0 when no pass ran.
• break ends the innermost loop and continue starts its next pass; break 2 and continue 2 act on the loop around it, and while : with a break is the shell’s loop whose test sits at the end.
• A retry loop needs a limit, a pause that grows, no pause after the last attempt and the last attempt’s status at the end, which until "$@" with an exit inside the body gives; a polling loop adds a deadline such as one from SECONDS.
• select prints a numbered menu on standard error and reads a line from standard input into REPLY: a listed number sets its variable to that word, any other text empties it, an empty line brings the menu back, and the loop goes on until break, in Bash but not in POSIX sh or dash.