Returning values from functions

A function that has worked out an answer has nowhere to put it: return carries a number from 0 to 255 and nothing else, so verdict=$(grade 55) leaves verdict empty even though the function ran and succeeded. Text and numbers reach the caller on other channels, and the channel you pick decides something bigger than syntax: whether the function may also change the caller’s variables, or runs in a copy of the shell whose changes are thrown away.

You will capture a function’s output and read its status, watch a counter rise inside that copy and fall back to its old value, write results into variables the caller names with printf -v and a namerefnameref: A variable made with declare -n or local -n that stands for another variable, so every use of it reads or changes the variable it names. More in “Indirection, namerefs and ${var@Q} transforms”, keep progress messages out of the captured data, and decide which channel fits a function of your own.

return hands back a number, so data leaves on standard output

return ends a function and sets its exit statusexit status: The number from 0 to 255 every command ends with, kept by the shell in $?. 0 means success; Bash uses 126 for found but not executable and 127 for not found. More in “Your first script: shebang, chmod and running it”, the number the functions lesson used to say whether a check had passed. A status is one small number, so there is no room in it for a word, a line or a list. The session below calls grade, which returns 0 for a pass, and then captures the same call: the status is 0 both times, and the captured value is empty, because the function printed nothing at all.

A function that is meant to hand data back prints it instead. initials prints two characters, and who=$(initials Asha Mehta) stores them: a command substitutioncommand substitution: $(command), or the older backquoted form: Bash runs the command in a subshell and puts its standard output in place of the substitution, with every trailing newline removed. More in “Command and process substitution” replaces the call with whatever the function wrote on its standard outputstandard output: Descriptor 1, where a command writes its results: the terminal, unless > file, >> file or a pipe points it somewhere else. Also called stdout. More in “Redirection: stdin, stdout, stderr, /dev/null”. Watch both halves, and the third pair of lines where a value and a status arrive from one call:

from session import run

run("""
# return ends the function with a number, so there is nothing to capture
grade() { [ "$1" -ge 40 ] && return 0; return 1; }
grade 55; echo "status $?"
verdict=$(grade 55); echo "verdict=[$verdict] status $?"
# a function that PRINTS its answer can be captured
initials() { printf '%s%s' "${1:0:1}" "${2:0:1}"; }
who=$(initials Asha Mehta); echo "who=$who"
# the value and the status arrive together
digits() { case $1 in *[!0-9]*) return 1 ;; esac; printf '%s' "${#1}"; }
n=$(digits 560034); echo "n=$n status $?"
!n=$(digits 56A034)
echo "n=[$n] status $?"
""")

# return ends the function with a number, so there is nothing to capture
$ grade() { [ "$1" -ge 40 ] && return 0; return 1; }
$ grade 55; echo "status $?"
status 0
$ verdict=$(grade 55); echo "verdict=[$verdict] status $?"
verdict=[] status 0
# a function that PRINTS its answer can be captured
$ initials() { printf '%s%s' "${1:0:1}" "${2:0:1}"; }
$ who=$(initials Asha Mehta); echo "who=$who"
who=AM
# the value and the status arrive together
$ digits() { case $1 in *[!0-9]*) return 1 ;; esac; printf '%s' "${#1}"; }
$ n=$(digits 560034); echo "n=$n status $?"
n=6 status 0
$ n=$(digits 56A034)
[exit status 1]
$ echo "n=[$n] status $?"
n=[] status 1

digits counts the characters of its argument, but only when every one of them is a digit: n=$(digits 560034) printed 6 with status 0, and n=$(digits 56A034) left n empty with status 1. The two channels are independent, so one call can say both “here is the answer” and “this did not work”.

One detail of capturing is worth knowing before you rely on it: a command substitution deletes every newline at the end of the output, so a function that prints a file cannot hand back the blank lines it ends with. The command and process substitution lesson works through that rule and the sentinel trick that gets around it.

A capture runs the function in a copy of the shell

Printing has a price. The manual says a command substitution runs its command in a subshell, and a subshellsubshell: A copy of the shell that runs commands apart from it, as Bash does for each command of a pipeline. It starts with the shell's variables, and every change it makes to them is lost when it exits. More in “Reading files line by line with read” is a copy of the shell with variables of its own. The function really runs, and what it changes really changes, but in the copy. When the copy ends, the variables it set and the folder it moved to end with it; only a file it wrote stays, because the file system is shared.

bump adds one to tally and prints what it saw. Called inside a capture it printed “bump saw 3”; afterwards the caller still had tally=2. Called plainly, the same function left tally=3. BASH_SUBSHELL, which Bash keeps as the number of subshells the current command sits inside, said why: 0 in the caller and 1 inside the capture. A cd goes the same way, as the last pair of lines shows:

from session import run

run("""
# a counter the function raises, read back by the caller
tally=2
bump() { tally=$(( tally + 1 )); echo "bump saw $tally"; }
note=$(bump); echo "$note"; echo "the caller still has tally=$tally"
bump; echo "after a plain call, tally=$tally"
# why: BASH_SUBSHELL counts the copies of the shell around the command
depth() { echo "running at depth $BASH_SUBSHELL"; }
depth
d=$(depth); echo "$d"
# the folder a capture moves to is lost as well
mkdir reports
enter() { cd reports; pwd; }
there=$(enter); echo "the function reached $there"; echo "the caller is in $PWD"
enter > /dev/null; echo "after a plain call, the caller is in $PWD"
""")

# a counter the function raises, read back by the caller
$ tally=2
$ bump() { tally=$(( tally + 1 )); echo "bump saw $tally"; }
$ note=$(bump); echo "$note"; echo "the caller still has tally=$tally"
bump saw 3
the caller still has tally=2
$ bump; echo "after a plain call, tally=$tally"
bump saw 3
after a plain call, tally=3
# why: BASH_SUBSHELL counts the copies of the shell around the command
$ depth() { echo "running at depth $BASH_SUBSHELL"; }
$ depth
running at depth 0
$ d=$(depth); echo "$d"
running at depth 1
# the folder a capture moves to is lost as well
$ mkdir reports
$ enter() { cd reports; pwd; }
$ there=$(enter); echo "the function reached $there"; echo "the caller is in $PWD"
the function reached ~/demo/reports
the caller is in ~/demo
$ enter > /dev/null; echo "after a plain call, the caller is in $PWD"
after a plain call, the caller is in ~/demo/reports

Follow the call once more, this time as a sequence between the caller and the copy, and note what crosses back:

Only two things come back from the copy: the bytes it printed and its exit status. That is the whole contract, and it explains a long list of surprises, among them a function whose counter never grows, a list that never gains an item and a read whose answer is gone by the next line. The reading files line by line lesson shows the same loss with a pipe into while, which is a subshell for the same reason.

An output parameter lets the caller name the variable to fill

The way to keep a function’s work is to let it run in the caller’s own shell and have it write the answer where the caller can read it. An output parameteroutput parameter: An argument that names the variable a function must write its result into, so the caller receives the value without capturing the function's output in a subshell. does that: the caller passes the name of a variable, and the function writes into it.

printf -v name '%s' "$value" is the portable-in-Bash form, there since bash-3.1, and from bash-4.1 it can write into an array element too. A nameref, local -n out=$1, makes the alias explicit: assignments to out land in the variable named by $1. The indirection and namerefs lesson covers namerefs in full, including the mistake that bites when the caller passes the nameref’s own name. The third form needs no argument at all: the function sets an agreed global variableglobal variable: A shell variable that no running function declared local, so every function of the script can read it and change it, and its value outlives the call that set it., which is what Bash’s own read does with REPLY.

three-ways.sh answers one question, “what is the value of this key?”, through all three channels, so the three calls can be compared line by line:

#!/usr/bin/env bash
# Three ways for a function to hand back one value: print it, write it with printf -v, write it through a nameref.
# Each one answers the same question: what is the value of this key in the settings text?

settings='retries=4
timeout=30
region=south'

# 1. Print the value; the caller captures the output.
value_of() {
  local line
  while IFS= read -r line; do
    case $line in
      "$1="*) printf '%s' "${line#*=}"; return 0 ;;
    esac
  done <<< "$settings"
  return 1
}

# 2. Write the value into the variable named by the first argument (printf -v, Bash 3.1 and later).
value_into() {
  local line
  while IFS= read -r line; do
    case $line in
      "$2="*) printf -v "$1" '%s' "${line#*=}"; return 0 ;;
    esac
  done <<< "$settings"
  return 1
}

# 3. The same, through a nameref (Bash 4.3 and later).
# shellcheck disable=SC2034  # out is a nameref: assigning it writes the variable the caller named
value_ref() {
  local -n out=$1
  local line
  while IFS= read -r line; do
    case $line in
      "$2="*) out=${line#*=}; return 0 ;;
    esac
  done <<< "$settings"
  return 1
}

main() {
  local printed written referred
  printed=$(value_of timeout)
  printf 'printed:  timeout=%s\n' "$printed"

  value_into written region
  printf 'printf -v: region=%s\n' "$written"

  value_ref referred retries
  printf 'nameref:  retries=%s\n' "$referred"

  if printed=$(value_of proxy); then
    printf 'printed:  proxy=%s\n' "$printed"
  else
    printf 'printed:  no proxy key (status %d)\n' "$?"
  fi
}

main "$@"

The run shows each form filling a variable, and the last call shows the printing form reporting a key that is not there. The commands after it do the same thing at the prompt, and the final line marks the limit of printf -v:

from session import files, run

files("three-ways.sh")
run("""
bash three-ways.sh
shellcheck three-ways.sh; echo "shellcheck said nothing, status $?"
# the same two writers, typed at the prompt: no capture, no subshell
fill_in() { printf -v "$1" '%s' "$2"; }
fill_in city Chennai; echo "city=$city"
hold() { local -n slot=$1; slot=$2; }
hold state Tamil-Nadu; echo "state=$state"
# a writer can report how its own work went, because its status is free
fill_digits() { case $2 in *[!0-9]*) return 1 ;; esac; printf -v "$1" '%s' "$2"; }
fill_digits pin 600001 && echo "pin=$pin"
!fill_digits pin 60O001
echo "pin is still $pin, status was 1"
# printf -v is Bash only: dash has no such option
!dash -c 'printf -v x "%s" hi'
""")

$ bash three-ways.sh
printed:  timeout=30
printf -v: region=south
nameref:  retries=4
printed:  no proxy key (status 1)
$ shellcheck three-ways.sh; echo "shellcheck said nothing, status $?"
shellcheck said nothing, status 0
# the same two writers, typed at the prompt: no capture, no subshell
$ fill_in() { printf -v "$1" '%s' "$2"; }
$ fill_in city Chennai; echo "city=$city"
city=Chennai
$ hold() { local -n slot=$1; slot=$2; }
$ hold state Tamil-Nadu; echo "state=$state"
state=Tamil-Nadu
# a writer can report how its own work went, because its status is free
$ fill_digits() { case $2 in *[!0-9]*) return 1 ;; esac; printf -v "$1" '%s' "$2"; }
$ fill_digits pin 600001 && echo "pin=$pin"
pin=600001
$ fill_digits pin 60O001
[exit status 1]
$ echo "pin is still $pin, status was 1"
pin is still 600001, status was 1
# printf -v is Bash only: dash has no such option
$ dash -c 'printf -v x "%s" hi'
dash: 1: printf: Illegal option -v
[exit status 2]

Three facts from that run are worth keeping. A writer keeps its status for itself, so fill_digits could refuse 60O001 with status 1 and leave the old value in place. ShellCheck does not follow a nameref, so without the directive above value_ref it reports the assignment to out as SC2034, “appears unused”; its page calls that an intended decision for variables reached only through indirection, and the directive’s comment says why the warning does not apply here. And printf -v is Bash’s own: dash answered “printf: Illegal option -v” with status 2.

Which of the four channels you can reach for depends on the oldest shell your script has to run on, so the table lists each one with the release that brought it and the call it leads to:

The last row is the one to watch: a global needs no argument, which is why it is tempting, and gives every caller the same name, which is why a nested call can overwrite a value its caller had not read yet.

Data on standard output, messages on standard error

Once standard output is the channel for data, anything else printed there becomes part of the data. A progress line, a warning, a “reading file…” note: each one is collected by the caller’s capture and arrives glued to the value.

stderr-logs.sh counts the lines of a log file twice. count_loud prints its progress line normally; count_quiet sends the same line to standard errorstandard error: Descriptor 2, where a command writes its warnings and error messages, kept apart from its results so that they still show when standard output goes to a file or a pipe. Also called stderr. More in “Redirection: stdin, stdout, stderr, /dev/null” with >&2. Both functions end by printing the count, and both are captured the same way:

#!/usr/bin/env bash
# A function whose progress line was captured together with its answer, and the same function with the line sent to
# standard error instead. Both report the number of lines in a log file.

# Writes its progress line on standard output, where a capture collects it with the answer.
count_loud() {
  echo "reading $1"
  wc -l < "$1" | tr -d ' '
}

# Writes its progress line on standard error, which no capture collects.
count_quiet() {
  echo "reading $1" >&2
  wc -l < "$1" | tr -d ' '
}

main() {
  local loud quiet
  loud=$(count_loud "$1")
  printf 'loud:  [%s]\n' "$loud"
  if [ "$loud" -gt 1 ]; then
    printf 'loud:  more than one line\n'
  else
    printf 'loud:  the test could not read it as a number\n'
  fi

  quiet=$(count_quiet "$1")
  printf 'quiet: [%s]\n' "$quiet"
  if [ "$quiet" -gt 1 ]; then
    printf 'quiet: more than one line\n'
  fi
}

main "$@"

Read the two values the run prints, and the test that follows each of them:

from session import files, run, write

files("stderr-logs.sh")
write("app.log", "start\nERROR disk full\nretry\ndone\n")
run("""
bash stderr-logs.sh app.log
# the message still reaches the terminal, and 2> /dev/null shows what the data channel carries
bash stderr-logs.sh app.log 2> /dev/null
# a function can report progress and hand back a value at the same time
pick() { echo "looking for $1" >&2; grep -c "$1" app.log; }
hits=$(pick ERROR); echo "hits=$hits"
echo "the line above came from the terminal, not from hits"
""")

$ bash stderr-logs.sh app.log
loud:  [reading app.log
4]
stderr-logs.sh: line 21: [: reading app.log
4: integer expected
loud:  the test could not read it as a number
reading app.log
quiet: [4]
quiet: more than one line
# the message still reaches the terminal, and 2> /dev/null shows what the data channel carries
$ bash stderr-logs.sh app.log 2> /dev/null
loud:  [reading app.log
4]
loud:  the test could not read it as a number
quiet: [4]
quiet: more than one line
# a function can report progress and hand back a value at the same time
$ pick() { echo "looking for $1" >&2; grep -c "$1" app.log; }
$ hits=$(pick ERROR); echo "hits=$hits"
looking for ERROR
hits=1
$ echo "the line above came from the terminal, not from hits"
the line above came from the terminal, not from hits

The loud version put “reading app.log” and the number 4 into one variable, and the arithmetic test on it failed with “integer expected” instead of comparing anything. The quiet version gave the clean value 4, and the note still appeared on the terminal, because standard error was never captured: that is the point of having two channels rather than one. Running the script with 2> /dev/null makes it plainer still, as the second call shows: the notes vanish and both values stay exactly as they were.

So a function that hands data back keeps a simple rule: the result goes to standard output, everything a person needs to read goes to standard error. The redirection lesson covers the descriptors behind >&2.

A function can hand back a value and a status at once

A capture used as an assignment keeps the status of the command inside it, so one line can collect a value and test whether the call worked. if p=$(port_of https); then printed the port, and if p=$(port_of gopher) took the other branch and reported status 1. This is the shape to reach for whenever a lookup may fail: no sentinel value to invent, no output to parse.

One builtinbuiltin: A command that is part of the shell itself rather than a separate program, such as cd, echo or type in Bash. help NAME documents it. More in “Commands, options, arguments and help” breaks it. local, like declare and export, has a status of its own, and on a line such as local p=$(port_of "$1") that status is what the caller sees, so a failed call reads as a success. Declaring the variable first keeps the two apart. The session runs both forms and then lets ShellCheck read the file:

from session import run, write

write("report.sh", """#!/usr/bin/env bash
port_of() { case $1 in http) echo 80 ;; https) echo 443 ;; *) return 1 ;; esac; }

one_line() {
  local p=$(port_of "$1")
  echo "one line: status $?, port [$p]"
}

two_lines() {
  local p
  p=$(port_of "$1")
  echo "two lines: status $?, port [$p]"
}

one_line gopher
two_lines gopher
""")
run("""
port_of() { case $1 in http) echo 80 ;; https) echo 443 ;; *) return 1 ;; esac; }
if p=$(port_of https); then echo "https uses port $p"; fi
if p=$(port_of gopher); then echo "port $p"; else echo "no port for that name, status $?"; fi
bash report.sh
!shellcheck report.sh
""")

$ port_of() { case $1 in http) echo 80 ;; https) echo 443 ;; *) return 1 ;; esac; }
$ if p=$(port_of https); then echo "https uses port $p"; fi
https uses port 443
$ if p=$(port_of gopher); then echo "port $p"; else echo "no port for that name, status $?"; fi
no port for that name, status 1
$ bash report.sh
one line: status 0, port []
two lines: status 1, port []
$ shellcheck report.sh

In report.sh line 5:
  local p=$(port_of "$1")
        ^-- SC2155 (warning): Declare and assign separately to avoid masking return values.

For more information:
  https://www.shellcheck.net/wiki/SC2155 -- Declare and assign separately to ...
[exit status 1]

one_line reported status 0 for a call that had failed; two_lines, with local p on its own line, reported 1. ShellCheck marked the first one as SC2155, “Declare and assign separately to avoid masking return values”, and exited

1. The command and process substitution lesson treats this rule and the other statuses an assignment can leave.

With both channels understood, the choice between them is a short list of questions about your own function: which shell has to run it, how often it is called, and whether the value must keep its bytes. Answer them in the table and watch the options drop out:

A #!/bin/sh script keeps to a capture or a global, because printf -v and namerefs are Bash’s own; a writer starts no child process at all; and a global is the cheapest and the least careful, because every caller shares the one name.

Practise: writers that fill the caller’s variables

Write these three functions in your own shell without a single $( ) inside them, using printf -v or a nameref for every result, then run the sample calls and compare what you get:

1. ext_into VAR FILE puts the extension of a file name, the part after its last dot, into the variable named by VAR. ext_into e report.tar.gz must leave e as gz with status 0, and ext_into e README must give status 1 and leave e as it was.
2. max_into VAR ARRAYNAME reads an array the caller names and puts its largest value into the variable named by VAR. With marks=(63 91 48 77), max_into top marks must leave top as 91 with status 0, and an empty array must give status 1 and leave top as it was.
3. to_upper_into VAR TEXT puts TEXT in upper case into the variable named by VAR, so to_upper_into shout "chennai 600001" leaves shout as CHENNAI 600001.

Interview questions

Warm-up (fresher to 2–5 years): why can return not hand text back, and which channels does a function use for data?

return ends a function and sets its exit status, and a status is one small number, 0 to 255, so there is nowhere in it for a word or a line. A capture of such a function is empty: with grade 55 returning 0, verdict=$(grade 55) left verdict empty and the status 0. Data travels on other channels. The function can print it, and the caller collects the print with v=$(fn), which runs the function in a subshell. Or the caller names a variable and the function writes into it, with printf -v "$1" '%s' "$value" or a nameref, which costs no subshell and keeps every other change the function makes. An agreed global variable, often REPLY, is the same idea without the argument.

Two harder questions build on this. In one, a report that looks a value up for every row runs for minutes; in the other, a settings parser has to hand back three different results from one call. Try each before you open it:

Key takeaways

• return carries an exit status and nothing else, so a function hands data back by printing it, and the caller collects that print with a command substitution while still reading the status of the call.
• A capture runs the function in a subshell, a copy of the shell: only the printed bytes and the status cross back, so a counter the function raises or a folder it changes to is lost when the copy ends.
• An output parameter keeps the work in the caller’s shell: printf -v "$1" '%s' "$value" from bash-3.1, a local -n nameref from bash-4.3, or an agreed global such as REPLY, and the function’s status is left for reporting how the call went.
• Standard output belongs to the data: send progress lines and warnings to standard error with >&2, or a capture collects them along with the value and the next test on it fails.
• An assignment from a capture keeps the status of its command, which is why if v=$(fn); then works, and why local v=$(fn) hides a failure behind the status of local, the slip ShellCheck reports as SC2155.