How Bash reads a line: tokens and expansions

n=3; echo {1..$n} looks as if it should print 1 2 3, yet Bash 5.3.20 prints {1..3}. Nothing is broken: Bash expands a line in a fixed order, and the braces are handled before the variable gets its value.

You will follow one line through every step Bash takes, from tokens to quote removal, predict which words get split and globbed and which stay whole, and use -x to see the exact words a command receives instead of guessing.

Bash cuts a line into tokens before it expands anything

Before any $ means anything, Bash reads the line and cuts it into tokenstoken: One piece of a command line as Bash first cuts it up: a word, or an operator such as ;, |, && or >. Operators are found before any expansion happens.: words, and operators such as ;, |, && and >. The Bash manual’s “Shell Operation” lists the steps that follow: the tokens are parsed into commands, each command’s words are expanded, redirections are set up, and the command runs, with the shell waiting for it to finish unless it was started in the background. So in n=3; echo {x,y}-$n *.txt, the ; ends the first command before either one is expanded, and a ; that appears later from a variable’s value is plain text, never an operator. Quoting acts at this stage too: a ; or | inside quotes is part of a word, so echo 'a;b' prints a;b instead of running a second command.

Expansion itself is a series of steps, always in the same order. Step through the line from the session in the next section and watch its words change at each step:

The figure follows the order the manual gives: brace expansion first; then tilde, parameter, arithmetic expansion and command substitution, together and left to right; then word splitting; then filename expansion; and finally quote removal. Process substitution, <(…), happens at the same time as the left-to-right group.

Because operators are found first, a ; stored in a variable cannot start a second command. This session tries it:

from session import run

# Operators are found when the line is cut into tokens, before expansion: a ; that comes out of a variable is only
# text, so Bash looks for a command whose name is "ls;", and notes.txt is never removed.
run("""
touch notes.txt
cmd='ls; rm notes.txt'
!$cmd
ls
""")

$ touch notes.txt
$ cmd='ls; rm notes.txt'
$ $cmd
bash: ls;: command not found
[exit status 127]
$ ls
notes.txt

Expansions run in a fixed order, so braces cannot use a variable

Here is the line from the figure, run by a Bash started with -x, which prints each command after expansion, with a + in front, before running it. The session sets HOME to /home/asha so the tilde’s result is visible:

from session import run

# One line that uses five kinds of expansion, run by a Bash started with -x, which prints each command after
# expansion (with "+ " in front) before running it.
run("""
touch notes.txt todo.txt
HOME=/home/asha
bash -xc 'n=3; echo {x,y}-$n ~/bin $((n * 2)) $(printf sub) *.txt'
""")

$ touch notes.txt todo.txt
$ HOME=/home/asha
$ bash -xc 'n=3; echo {x,y}-$n ~/bin $((n * 2)) $(printf sub) *.txt'
+ n=3
++ printf sub
+ echo x-3 y-3 /home/asha/bin 6 sub notes.txt todo.txt
x-3 y-3 /home/asha/bin 6 sub notes.txt todo.txt

The trace shows the seven words echo received. The line ++ printf sub comes first because the command substitution runs, in a subshell of its own, while the line is being expanded; the doubled plus marks that deeper level. Brace expansion happens before everything else, which explains the puzzle at the top of this lesson:

from session import run

# Brace expansion runs before parameter expansion, so a variable cannot set a range; two ways that work, and dash.
run("""
n=3
echo {1..3}
echo {1..$n}
seq 1 "$n"
for ((i = 1; i <= n; i++)); do printf '%s ' "$i"; done; echo
dash -c 'echo {1..3}'
""")

$ n=3
$ echo {1..3}
1 2 3
$ echo {1..$n}
{1..3}
$ seq 1 "$n"
1
2
3
$ for ((i = 1; i <= n; i++)); do printf '%s ' "$i"; done; echo
1 2 3
$ dash -c 'echo {1..3}'
{1..3}

When the braces are expanded, {1..$n} is not a valid range, so it stays as it is; then $n becomes 3, giving the literal text {1..3}. For a range set by a variable, use seq 1 "$n" or a C-style for ((i = 1; i <= n; i++)) loop. The last command shows that dash, a plain POSIX shell, has no brace expansion at all.

Process substitution belongs to the left-to-right group too. In diff <(sort a.txt) <(sort b.txt) each <(…) runs its command and is replaced by a file name, such as /dev/fd/63, from which the command’s output can be read, so diff compares two sorted files that never exist on disk.

Only unquoted results are split and globbed

Word splitting and filename expansion do not touch everything. They apply to the unquoted results of parameter expansion, command substitution and arithmetic expansion; text written in quotes, and words you typed yourself, are never split again. printf '[%s]\n' prints each word it receives in brackets, which makes the difference visible:

from session import run

# What a command really receives: printf '[%s]' prints each word it was given in brackets. Unquoted, the value of x is
# split and its *.txt is globbed; quoted, it stays one word. An assignment written before a command applies to that
# command only. A redirection's target is expanded too, and must stay one word.
run("""
touch one.txt two.txt
x='a  b   *.txt'
printf '[%s]\\n' $x
printf '[%s]\\n' "$x"
printf '[%s]\\n' a  b   '*.txt'
GREETING=hello bash -c 'printf "%s\\n" "$GREETING"'
printf '[%s]\\n' "$GREETING"
out='my words.txt'
!printf '%s\\n' hi > $out
printf '%s\\n' hi > "$out"
ls
""")

$ touch one.txt two.txt
$ x='a  b   *.txt'
$ printf '[%s]\n' $x
[a]
[b]
[one.txt]
[two.txt]
$ printf '[%s]\n' "$x"
[a  b   *.txt]
$ printf '[%s]\n' a  b   '*.txt'
[a]
[b]
[*.txt]
$ GREETING=hello bash -c 'printf "%s\n" "$GREETING"'
hello
$ printf '[%s]\n' "$GREETING"
[]
$ out='my words.txt'
$ printf '%s\n' hi > $out
bash: $out: ambiguous redirect
[exit status 1]
$ printf '%s\n' hi > "$out"
$ ls
my words.txt
one.txt
two.txt

Unquoted, the value of x became four words: a, b, and the two file names that *.txt matched. Quoted, it stayed one word with its spaces and its asterisk. The third command splits on spaces too, but that is the line being cut into tokens, and '*.txt' was quoted, so it is passed as it is. The two GREETING commands show an assignment written before a command name: GREETING=hello reached the one command it was written before, and the shell itself never set it.

The last three commands show where redirections fit. The manual sets them up after the words are expanded, and the word after > is expanded like any other: > $out became two words, my and words.txt, and Bash refused with ambiguous redirect, because a redirection needs exactly one file name. Quoted, > "$out" created my words.txt, as ls shows. Splitting therefore causes errors in redirections as well as wrong arguments, and the same double quotes cure both.

A trace shows the words a command really receives

Tracing turns the guesswork into a record. bash -x script.sh runs a script with tracing on, and inside a script or a shell set -x turns it on and set +x off. Each traced line is the command after every expansion, with quotes added wherever a word contains spaces, so you see where one word ends and the next begins:

from session import run, write

# bash -x prints every command after expansion, prefixed with PS4 ("+ " by default), before running it; a PS4 with
# $LINENO says which line of the script each trace comes from.
write("greet.sh", '#!/usr/bin/env bash\nname="Asha Rao"\nprintf "[%s]\\n" $name\nprintf "[%s]\\n" "$name"\n')
run("""
cat greet.sh
bash -x greet.sh
PS4='+ line $LINENO: ' bash -x greet.sh
""")

$ cat greet.sh
#!/usr/bin/env bash
name="Asha Rao"
printf "[%s]\n" $name
printf "[%s]\n" "$name"
$ bash -x greet.sh
+ name='Asha Rao'
+ printf '[%s]\n' Asha Rao
[Asha]
[Rao]
+ printf '[%s]\n' 'Asha Rao'
[Asha Rao]
$ PS4='+ line $LINENO: ' bash -x greet.sh
+ line 2: name='Asha Rao'
+ line 3: printf '[%s]\n' Asha Rao
[Asha]
[Rao]
+ line 4: printf '[%s]\n' 'Asha Rao'
[Asha Rao]

The first trace shows printf '[%s]\n' Asha Rao: two separate words, because $name was unquoted, and the second shows 'Asha Rao' as one. The prefix is the variable PS4, + by default; setting it to include $LINENO names the line of the script each trace comes from, which helps in a script of a few hundred lines.

Two related tools are worth knowing. set -v prints each line as it is read, before any expansion, so with both -v and -x you see the line as written and then as expanded. And because traces go to standard error, a script’s real output on standard output stays clean; bash -x script.sh 2> trace.log keeps the whole trace in a file to read afterwards.

Practise: predicting what each line becomes

The questions show lines like the ones in the sessions and ask what each command receives. Each choice is explained as soon as you answer, and the step you get wrong most often comes back:

Interview questions

Warm-up (fresher to 2–5 years): in what order does Bash expand a command line?

Bash first cuts the line into tokens, words and operators, and parses it into commands. Then it expands each word in a fixed order: brace expansion; then tilde expansion, parameter and variable expansion, arithmetic expansion and command substitution, all together from left to right, with process substitution at the same time; then word splitting of the unquoted results; then filename expansion; and quote removal last. The order explains real behaviour: {1..$n} stays literal because braces are expanded before $n has a value, and a ; inside a variable’s value is never an operator because operators were found before expansion.

Two harder questions follow, about a command stored in a variable and about a script that breaks on file names with spaces. Work out your answer, then open each first step:

Key takeaways

• Bash cuts a line into tokens and commands before it expands anything, so operators that come from a variable’s value are plain text.
• Every word is expanded in one order: braces; tilde, variables, arithmetic and command substitution from left to right; then word splitting, filename expansion and quote removal.
• Only the unquoted results of expansions are split and globbed, which is why quoting a variable keeps its value as one word.
• bash -x and set -x print each command after expansion, prefixed with PS4, so you can see the exact words a command receives.