Redirection: stdin, stdout, stderr, /dev/null

./report.sh > out.txt saved two of the report’s three lines in out.txt and still printed the third, a warning, on the screen, because the warning travelled on a different stream. Every command starts with three: standard inputstandard input: Descriptor 0, the stream a command reads from: the keyboard in a terminal, unless < file or a pipe points it somewhere else. Also called stdin. (stdin), where it reads, by default from the keyboard; standard outputstandard output: Descriptor 1, where a command writes its results: the terminal, unless > file, >> file or a pipe points it somewhere else. Also called stdout. (stdout), where its results go; and standard errorstandard error: Descriptor 2, where a command writes its warnings and error messages, kept apart from its results so that they still show when standard output goes to a file or a pipe. Also called stderr. (stderr), where its warnings and errors go. In a terminal, stdout and stderr both go to the screen until a redirectionredirection: Notation such as > file, 2> file, < file or 2>&1 that points one of a command's file descriptors at a file, or at the place another descriptor points to, before the command runs. Redirections are applied from left to right. points one of them somewhere else, and /dev/nullnull device: /dev/null, a device file that discards everything written to it and gives end of file at once to anything that reads it. Pointing a stream at it throws that output away. is the place to point a stream you want thrown away: whatever is written there disappears.

You will send results and errors to separate files or to one, predict what each order of 2>&1 and > file does when the output goes on into a pipeA one-way channel between two commands, written cmd1 | cmd2: what cmd1 writes on its standard output, cmd2 reads on its standard input., recognise the &> line that dash runs in the background, stop > from overwriting a file, and keep a log with tee while you watch the output.

Every command starts with three descriptors: 0, 1 and 2

A file descriptorfile descriptor: A small number a running program uses to name one of its open files or streams. A program normally starts with 0 (standard input), 1 (standard output) and 2 (standard error) already open., fd for short, is a small number a running program uses to name one of its open files. A program normally starts with three of them open, and POSIX fixes their numbers: 0 is stdin, 1 is stdout and 2 is stderr. A program writes its results to descriptor 1 and its complaints to descriptor 2 without knowing where either goes. The session’s report.sh writes two results with plain echo and one warning with echo … >&2, which points that one echo’s descriptor 1 at whatever descriptor 2 points to:

from session import run, write

write("report.sh", """#!/usr/bin/env bash
# report.sh: two results on standard output, one warning on standard error
echo "orders read: 3"
echo "warning: order 17 has no price" >&2
echo "total: 4200"
""")
run("""
cat report.sh
chmod +x report.sh
./report.sh
./report.sh > out.txt
cat out.txt
./report.sh 2> err.txt
cat err.txt
!tr a-z A-Z out.txt
tr a-z A-Z < out.txt
./report.sh >> out.txt 2>> err.txt
wc -l out.txt err.txt
./report.sh > out.txt 2> err.txt
wc -l out.txt err.txt
echo "saved" 2 > note.txt; cat note.txt
!cat < missing.txt
!./report.sh > nodir/out.txt
""")

$ cat report.sh
#!/usr/bin/env bash
# report.sh: two results on standard output, one warning on standard error
echo "orders read: 3"
echo "warning: order 17 has no price" >&2
echo "total: 4200"
$ chmod +x report.sh
$ ./report.sh
orders read: 3
warning: order 17 has no price
total: 4200
$ ./report.sh > out.txt
warning: order 17 has no price
$ cat out.txt
orders read: 3
total: 4200
$ ./report.sh 2> err.txt
orders read: 3
total: 4200
$ cat err.txt
warning: order 17 has no price
$ tr a-z A-Z out.txt
usage: tr [-Ccsu] string1 string2
       tr [-Ccu] -d string1
       tr [-Ccu] -s string1
       tr [-Ccu] -ds string1 string2
[exit status 1]
$ tr a-z A-Z < out.txt
ORDERS READ: 3
TOTAL: 4200
$ ./report.sh >> out.txt 2>> err.txt
$ wc -l out.txt err.txt
       4 out.txt
       2 err.txt
       6 total
$ ./report.sh > out.txt 2> err.txt
$ wc -l out.txt err.txt
       2 out.txt
       1 err.txt
       3 total
$ echo "saved" 2 > note.txt; cat note.txt
saved 2
$ cat < missing.txt
bash: missing.txt: No such file or directory
[exit status 1]
$ ./report.sh > nodir/out.txt
bash: nodir/out.txt: No such file or directory
[exit status 1]

Run with no redirection, the three lines looked alike. > out.txt caught the two results and left the warning on the screen; 2> err.txt caught the warning alone. The operators differ by the descriptor they change and by what they do to the file:

Standard input is where a filter such as sort, wc or grep reads when it is given no file name: the keyboard in a terminal, unless < or a pipe points descriptor 0 at something else. tr reads nothing else, so tr a-z A-Z out.txt failed with a usage message and status 1, and tr a-z A-Z < out.txt handed it the file as descriptor 0. One more run with >> and 2>> grew the two files from 2 and 1 lines to 4 and 2; the next run with > and 2> cut them back to 2 and 1, because > empties the file every time it opens it.

The shell carries out every redirection before the command starts. cat < missing.txt failed with a message from bash, not from cat, and ./report.sh > nodir/out.txt printed no warning at all: the folder did not exist, so the script never ran. Both ended with exit statusexit status: The number from 0 to 255 every command ends with, kept by the shell in $?. 0 means success; Bash uses 126 for found but not executable and 127 for not found. More in “Your first script: shebang, chmod and running it” 1.

Redirections run from left to right, and a copy keeps its old target

Sending both streams to one file takes two redirections: > out.txt for descriptor 1, and 2>&1, which makes descriptor 2 a copy of descriptor 1. Written in either order they look alike, and they are not. The session pipes each command into sed 's/^/pipe> /', which marks every line that reaches it, so each line shows which way it went:

from session import run, write

write("report.sh", """#!/usr/bin/env bash
# report.sh: two results on standard output, one warning on standard error
echo "orders read: 3"
echo "warning: order 17 has no price" >&2
echo "total: 4200"
""")
write("check.sh", """#!/usr/bin/env bash
# check.sh: run the report with its results and its warning in out.txt
./report.sh 2>&1 > out.txt
""")
run("""
chmod +x report.sh
./report.sh 2>&1 | sed 's/^/pipe> /'
./report.sh > out.txt 2>&1 | sed 's/^/pipe> /'
cat out.txt
./report.sh 2>&1 > out.txt | sed 's/^/pipe> /'
cat out.txt
!shellcheck check.sh
""")

$ chmod +x report.sh
$ ./report.sh 2>&1 | sed 's/^/pipe> /'
pipe> orders read: 3
pipe> warning: order 17 has no price
pipe> total: 4200
$ ./report.sh > out.txt 2>&1 | sed 's/^/pipe> /'
$ cat out.txt
orders read: 3
warning: order 17 has no price
total: 4200
$ ./report.sh 2>&1 > out.txt | sed 's/^/pipe> /'
pipe> warning: order 17 has no price
$ cat out.txt
orders read: 3
total: 4200
$ shellcheck check.sh

In check.sh line 3:
./report.sh 2>&1 > out.txt
            ^--^ SC2069 (warning): To redirect stdout+stderr, 2>&1 must be last (or use '{ cmd > file; } 2>&1' to clarify).

For more information:
  https://www.shellcheck.net/wiki/SC2069 -- To redirect stdout+stderr, 2>&1 m...
[exit status 1]

The first command is the baseline: 2>&1 alone sent the warning into the pipe with the results, and sed marked all three lines. With > out.txt 2>&1, nothing came through the pipe and out.txt held all three lines. With 2>&1 > out.txt, only the warning came through, as pipe> warning: order 17 has no price, and out.txt held the two results.

Three rules explain it. Bash connects the pipe before it applies any redirection, so descriptor 1 starts at the pipe. The shell then applies the redirections in the order they are written, from left to right, as the Bash manual and POSIX both state. And 2>&1 copies where descriptor 1 points at that moment; it does not tie descriptor 2 to descriptor 1 for later. Step through both orders, one redirection at a time:

In the first order the copy is made after descriptor 1 has moved to the file, so both streams end there. In the second it is made while descriptor 1 still points to the pipe, and moving descriptor 1 afterwards leaves descriptor 2 behind. Without a pipe, descriptor 1 starts at the terminal, and the second order leaves the warning on the screen. ShellCheck flags that order in a script as SC2069, “2>&1 must be last”. Its wiki adds that the warning stays silent when the output is piped, and there the second order has a real use: cmd 2>&1 > /dev/null | grep word searches the errors alone.

Bash’s short form for both streams breaks under dash

Bash has a shorter way to send both streams to one file: &> file means > file 2>&1, and &>> file means >> file 2>&1. The Bash manual also accepts >& file for the first, but prefers &>. In the session, &> both.txt caught all three lines, and &>> added three more, six in all:

from session import run, write

write("report.sh", """#!/usr/bin/env bash
# report.sh: two results on standard output, one warning on standard error
echo "orders read: 3"
echo "warning: order 17 has no price" >&2
echo "total: 4200"
""")
write("log.sh", """#!/bin/sh
# log.sh: a POSIX sh script that uses the Bash short form
./report.sh &> report.log
""")
run("""
chmod +x report.sh
./report.sh &> both.txt
cat both.txt
./report.sh &>> both.txt
wc -l both.txt
printf 'kept line\\n' > dash.txt
wc -c dash.txt
dash -c './report.sh &> dash.txt; wait'
wc -c dash.txt
!shellcheck log.sh
/bin/bash -c 'echo "$BASH_VERSION"'
/bin/bash -c './report.sh &> old.txt'; wc -l old.txt
!/bin/bash -c './report.sh &>> old.txt'
""")

$ chmod +x report.sh
$ ./report.sh &> both.txt
$ cat both.txt
orders read: 3
warning: order 17 has no price
total: 4200
$ ./report.sh &>> both.txt
$ wc -l both.txt
       6 both.txt
$ printf 'kept line\n' > dash.txt
$ wc -c dash.txt
      10 dash.txt
$ dash -c './report.sh &> dash.txt; wait'
orders read: 3
warning: order 17 has no price
total: 4200
$ wc -c dash.txt
       0 dash.txt
$ shellcheck log.sh

In log.sh line 3:
./report.sh &> report.log
            ^-----------^ SC3020 (warning): In POSIX sh, &> is undefined.

For more information:
  https://www.shellcheck.net/wiki/SC3020 -- In POSIX sh, &> is undefined.
[exit status 1]
$ /bin/bash -c 'echo "$BASH_VERSION"'
3.2.57(1)-release
$ /bin/bash -c './report.sh &> old.txt'; wc -l old.txt
       3 old.txt
$ /bin/bash -c './report.sh &>> old.txt'
/bin/bash: -c: line 0: syntax error near unexpected token `>'
/bin/bash: -c: line 0: `./report.sh &>> old.txt'
[exit status 2]

POSIX has no &> operator, and dash, which follows POSIX here, reads the two characters as two operators it knows. ./report.sh & is a command started in the background, and > dash.txt is a command with no name, only a redirection, which POSIX says is carried out and nothing else: the file is created, or emptied. So the report’s three lines went to the terminal, and dash.txt, which held 10 bytes, was left with 0. The session adds ; wait so that dash waits for the background job before the next command, which it otherwise would not do.

ShellCheck catches the line before it ever runs: in a script that starts with #!/bin/sh it reports SC3020, “In POSIX sh, &> is undefined”, and its wiki gives the replacement, > log 2>&1. That long form, and >> log 2>&1 for appending, mean the same in every POSIX shell, so they belong in any script that sh might run.

/dev/null throws output away, and a quiet grep needs none

Some output is noise: a progress message, a warning you have already dealt with, the matching lines of a search whose only question is whether anything matched. POSIX defines /dev/null as an empty data source and an endless sink: whatever is written to it is discarded, and reading it gives end of file at once. Point a stream at it to silence it:

from session import run, write

write("report.sh", """#!/usr/bin/env bash
# report.sh: two results on standard output, one warning on standard error
echo "orders read: 3"
echo "warning: order 17 has no price" >&2
echo "total: 4200"
""")
write("orders.log", "order 15 priced\norder 16 priced\norder 17 has no price\n")
run("""
chmod +x report.sh
./report.sh > /dev/null
./report.sh 2> /dev/null
./report.sh > /dev/null 2>&1
!ls missing.txt > /dev/null 2>&1
wc -c < /dev/null
grep 'no price' orders.log > /dev/null && echo "a price is missing"
grep -q 'no price' orders.log && echo "a price is missing"
yes 'order 18 has no price' | grep -q 'no price'; echo "grep -q returned $?"
!grep -q 'no price' missing.log
!grep -qs 'no price' missing.log
""")

$ chmod +x report.sh
$ ./report.sh > /dev/null
warning: order 17 has no price
$ ./report.sh 2> /dev/null
orders read: 3
total: 4200
$ ./report.sh > /dev/null 2>&1
$ ls missing.txt > /dev/null 2>&1
[exit status 1]
$ wc -c < /dev/null
       0
$ grep 'no price' orders.log > /dev/null && echo "a price is missing"
a price is missing
$ grep -q 'no price' orders.log && echo "a price is missing"
a price is missing
$ yes 'order 18 has no price' | grep -q 'no price'; echo "grep -q returned $?"
grep -q returned 0
$ grep -q 'no price' missing.log
grep: missing.log: No such file or directory
[exit status 2]
$ grep -qs 'no price' missing.log
[exit status 2]

> /dev/null silenced the results and left the warning, 2> /dev/null did the opposite, and > /dev/null 2>&1, in that order, silenced both. The text goes, the status stays: ls missing.txt > /dev/null 2>&1 printed nothing and still ended with status 1, so a script can test a command quietly. As input, /dev/null is empty: wc -c < /dev/null counted 0 bytes, an easy way to give a command nothing to read.

Searching with grep … > /dev/null works, but grep -q says what you mean. POSIX defines -q to write nothing to standard output and to exit with 0 when a line is selected, and notes that it can stop at the first match. That is why yes 'order 18 has no price' | grep -q 'no price' returned with status 0, although yes repeats its line for ever: grep stopped at the first matching line, and the next lesson shows how the pipeline then ends yes.

-q quiets standard output only. With missing.log the error still appeared, on standard error, with status 2, and -s, which POSIX defines to suppress the messages about missing or unreadable files, removed the message but kept the status.

The noclobber option refuses to overwrite a file

> empties a file without asking, so one wrong file name can wipe out a previous run’s results. The noclobbernoclobber: A shell option, switched on with set -C or set -o noclobber, that makes > refuse to overwrite an existing regular file; >| overrides it for one redirection. option, switched on with set -C (or set -o noclobber) and off with set +C, makes > refuse to open a regular file that already exists:

from session import run, write

write("totals.txt", "first run: 4200\n")
run("""
set -C
!echo "second run: 3900" > totals.txt
echo "second run: 3900" >> totals.txt
cat totals.txt
echo "third run: 5100" >| totals.txt
cat totals.txt
echo "draft" > new.txt
echo "noise" > /dev/null
echo "from tee" | tee totals.txt
cat totals.txt
set +C
""")

$ set -C
$ echo "second run: 3900" > totals.txt
bash: totals.txt: cannot overwrite existing file
[exit status 1]
$ echo "second run: 3900" >> totals.txt
$ cat totals.txt
first run: 4200
second run: 3900
$ echo "third run: 5100" >| totals.txt
$ cat totals.txt
third run: 5100
$ echo "draft" > new.txt
$ echo "noise" > /dev/null
$ echo "from tee" | tee totals.txt
from tee
$ cat totals.txt
from tee
$ set +C

The refused > printed “cannot overwrite existing file”, ended with status 1 and left totals.txt as it was, while >> still added a line, since appending destroys nothing. >| is the deliberate overwrite: it replaced both lines with the third run’s result. A new file was created without complaint, and the write to /dev/null went through too, since /dev/null is a device, not a regular file. tee totals.txt overwrote the file anyway: noclobber is a rule of the shell’s >, and tee opens its files itself.

POSIX adds a guarantee that makes the option useful beyond caution: the check that the file does not exist and its creation happen as one step, as when a program opens a file with the flags O_CREAT and O_EXCL. Two scripts that run set -C and then : > job.lock at the same moment cannot both succeed, which POSIX says is what makes lock files created this way reliable, with caveats it lists in its rationale.

tee copies one stream to files and to the screen

A long job raises a choice: watch its output or keep a log. tee does both. POSIX defines it to copy its standard input to its standard output and to every file it is given, emptying those files first, and -a makes it append to them instead:

from session import run, write

write("report.sh", """#!/usr/bin/env bash
# report.sh: two results on standard output, one warning on standard error
echo "orders read: 3"
echo "warning: order 17 has no price" >&2
echo "total: 4200"
""")
write("count.awk", 'BEGIN { print "line 1 to stdout"; print "line 2 to stderr" > "/dev/stderr"; print "line 3 to stdout" }\n')
run("""
chmod +x report.sh
./report.sh 2> err.txt | tee run.log
cat err.txt
./report.sh 2>&1 | tee run.log
cat run.log
./report.sh 2>&1 | tee -a run.log
wc -l run.log
./report.sh 2>&1 | tee -a run.log > /dev/null
wc -l run.log
awk -f count.awk 2>&1 | tee merged.log
""")

$ chmod +x report.sh
$ ./report.sh 2> err.txt | tee run.log
orders read: 3
total: 4200
$ cat err.txt
warning: order 17 has no price
$ ./report.sh 2>&1 | tee run.log
orders read: 3
warning: order 17 has no price
total: 4200
$ cat run.log
orders read: 3
warning: order 17 has no price
total: 4200
$ ./report.sh 2>&1 | tee -a run.log
orders read: 3
warning: order 17 has no price
total: 4200
$ wc -l run.log
       6 run.log
$ ./report.sh 2>&1 | tee -a run.log > /dev/null
$ wc -l run.log
       9 run.log
$ awk -f count.awk 2>&1 | tee merged.log
line 2 to stderr
line 1 to stdout
line 3 to stdout

2> err.txt | tee run.log showed and logged the two results and kept the warning in err.txt, because only descriptor 1 goes into a pipe. 2>&1 | tee run.log put all three lines on the screen and in the log. A run with tee -a made it 6 lines, and tee -a run.log > /dev/null added three more, 9 in all, without showing them: a quiet way for a script to keep its log. To split a pasted log the same way in a browser, the filter lines tool keeps the lines that contain a word, such as warning, beside the lines it removes.

The last command shows the limit of a merged log. The small awk program prints line 1 and line 3 to standard output and line 2 to standard error, in that order, yet the log begins with line 2. POSIX lets a program keep its standard output in a buffer when that output is not a terminal, and write it later in blocks, here after line 2 had already gone out, while standard error must never be buffered that way. A merged log keeps the order in which bytes were written, not the order of the program’s print statements; the order within each stream is always kept.

Practise: split a report into three logs

Real scripts often need all three logs at once: the results, the warnings, and both together. Build it in your own terminal with the report.sh of the first session:

1. Write a command, or two, that leaves the two results in out.log, the warning in err.log and all three lines in all.log, each file emptied first.
2. Check it: wc -l out.log err.log all.log must show 2, 1 and 3, grep -c warning out.log must print 0, and all.log must hold the results in their own order plus the warning.
3. Run it again and check that the counts stay 2, 1 and 3. Then change it to append, run it twice from empty files, and check for 4, 2 and 6.

One way runs the report once with > out.log 2> err.log and then builds all.log from the two files with cat; it keeps the order within each stream, which is all a merged log can promise. Doing it in a single command is easiest with a third descriptor or with process substitution, from File descriptors and exec redirections and Command and process substitution.

The questions below use new commands and files with the rules of the sessions. Work out where every line goes before you answer:

Interview questions

Warm-up: name the three standard streams with their numbers, and say what 2>&1 does (fresher to 2–5 years).

Standard input is descriptor 0, where a command reads, by default from the keyboard. Standard output is descriptor 1, for results, and standard error is descriptor 2, for warnings and errors; in a terminal both go to the screen. 2>&1 makes descriptor 2 a copy of descriptor 1 as it points at that moment, so errors go wherever results are going. Because redirections run from left to right, cmd > out.txt 2>&1 puts both streams in out.txt, while cmd 2>&1 > out.txt sends errors to the old standard output, the terminal or a pipe, and only the results to the file. Piped into sed, the second form let the warning alone through the pipe.

The two harder questions are about a scheduled job whose failures never reach its log and a sh script whose log stays empty on another machine. For each, say where every descriptor ends up before you read the first step on its page:

Key takeaways

• Every command starts with descriptor 0 for standard input, 1 for standard output and 2 for standard error; < reads a file on 0, > and 2> empty a file and write it, >> and 2>> append, and the shell opens each file before the command runs.
• Redirections are applied from left to right and 2>&1 copies where descriptor 1 points at that moment, so > file 2>&1 sends both streams to the file while 2>&1 > file leaves standard error on the old standard output, the terminal or a pipe.
• &> file and &>> file are Bash’s short forms of > file 2>&1 and >> file 2>&1; dash runs the command in the background and empties the file instead, and Bash 3.2 lacks &>>, so scripts that sh may run use the long forms.
• /dev/null discards what is written to it and reads as empty, a redirection to it hides text but never the exit status, and grep -q tests for a match without printing it and can stop at the first one.
• set -C makes > refuse to overwrite an existing regular file and >| overrides it for one redirection, while >>, new files and /dev/null still work, and tee, which opens its files itself, is not affected.
• tee copies its standard input to the screen and to files, emptying them unless -a appends, and a merged log keeps the order within each stream but not always the order across the two.