Globbing: wildcards, dotfiles, extglob, globstar

In a folder with no Markdown files, count *.md reports one word: the text *.md itself. A script that then runs rm *.md or loops over the result works on a file that does not exist, unless you choose what an unmatched pattern should do.

You will match files with the four basic patterns, decide what a pattern that matches nothing becomes, include hidden files on purpose, search folders with **, use extended patterns, and sort matches with Bash 5.3’s GLOBSORT.

Four pattern characters match file names, and an unmatched pattern stays as it is

After word splitting, Bash looks at each unquoted word for the characters *, ? and [. Such a word is a glob patternglob pattern: A word containing *, ? or [ that Bash replaces with the sorted names of the files it matches; it stays as written when nothing matches, unless nullglob or failglob is set., and filename expansion replaces it with the matching names, sorted. * matches any string, ? any one character, [ab] one of the listed characters and [!ab] any character not listed.

The session builds a folder with a.txt, b.txt, c.log, a hidden .hidden.txt and d e.txt, then counts what each pattern becomes:

from session import run

# A scratch folder with five files, one of them hidden and one with a space. count prints how many words a pattern
# became and each word in brackets, for the basic patterns and for the shell options that change what a pattern does
# when nothing, or a hidden file, matches.
run("""
touch a.txt b.txt c.log .hidden.txt 'd e.txt'
count() { printf '%d:' "$#"; (( $# )) && printf ' [%s]' "$@"; printf '\\n'; }
count *.txt
count ?.txt
count [ab].*
count [!ab]*
count *.md
shopt -s nullglob; count *.md; shopt -u nullglob
!shopt -s failglob; count *.md
shopt -u failglob
shopt -s dotglob; count *.txt; shopt -u dotglob
count .*
count *.TXT
shopt -s nocaseglob; count *.TXT; shopt -u nocaseglob
""")

$ touch a.txt b.txt c.log .hidden.txt 'd e.txt'
$ count() { printf '%d:' "$#"; (( $# )) && printf ' [%s]' "$@"; printf '\n'; }
$ count *.txt
3: [a.txt] [b.txt] [d e.txt]
$ count ?.txt
2: [a.txt] [b.txt]
$ count [ab].*
2: [a.txt] [b.txt]
$ count [!ab]*
2: [c.log] [d e.txt]
$ count *.md
1: [*.md]
$ shopt -s nullglob; count *.md; shopt -u nullglob
0:
$ shopt -s failglob; count *.md
bash: no match: *.md
[exit status 1]
$ shopt -u failglob
$ shopt -s dotglob; count *.txt; shopt -u dotglob
4: [.hidden.txt] [a.txt] [b.txt] [d e.txt]
$ count .*
1: [.hidden.txt]
$ count *.TXT
1: [*.TXT]
$ shopt -s nocaseglob; count *.TXT; shopt -u nocaseglob
3: [a.txt] [b.txt] [d e.txt]

*.txt matched 3 names, and d e.txt arrived as one word with its space: filename expansion happens after word splitting, so a matched name is never split. *.txt did not match .hidden.txt, because a leading dot is matched only by a pattern that starts with a dot. And *.md, matching nothing, stayed as the literal word *.md.

The order of the names comes from the locale’s collation. The session runs with LC_ALL=C.UTF-8, where names sort by their bytes, so capital letters come before small ones; in a language locale such as en_US.UTF-8 they may be interleaved. A script that depends on the order should set the locale it expects. To match a pattern character literally, quote it: '*'.txt or \*.txt matches only a file whose name really starts with an asterisk.

The same pattern language works outside filename expansion, in case and in [[ … ]]: case "$f" in *.txt) compares the word with the pattern without looking at any folder, and [[ $f == *.txt ]] does the same. There a pattern never turns into names; it only answers yes or no.

Globs are not regular expressions: in a glob * means “any string”, while in a regular expression it means “repeat the previous item”. *.txt as a regular expression would be an error or would mean something quite different.

Shell options decide what happens with no match and with hidden files

Bash’s shopt builtin switches options on with -s and off with -u, and four of them change filename expansion. In the session, nullglob made the unmatched *.md vanish, 0 words, and failglob made it an error, no match: *.md, with status 1. dotglob let *.txt match the hidden file as well, and nocaseglob let *.TXT match the three lower-case names that the plain pattern missed. Without it, .* matched only .hidden.txt: since Bash 5.2 the globskipdots option, on by default, keeps . and .. out of every match.

The options arrived over many releases, which matters when a script may meet an older Bash, such as the 3.2.57 that macOS ships as /bin/bash:

globstar makes ** match folders at any depth, and an older Bash treats ** like *. To see an option’s state before you rely on it, shopt nullglob prints nullglob off or nullglob on, and shopt -p prints every option as a command that sets it back. Options set with shopt stay set until the shell exits, so a script that needs one for a single command should switch it off again afterwards, as the session did after each test. Which no-match behaviour to choose depends on the script: nullglob suits loops that may have nothing to do, while failglob suits commands that should never run with a literal pattern, such as rm.

globstar and extglob search deeper and match anything but a pattern

Two options add new kinds of pattern. With globstar, **/*.log matches log files in the folder and every folder below it. With extglob, five patterns from the Korn shell become available: ?(…) zero or one, *(…) any number, +(…) one or more, @(…) exactly one, and !(…) anything except the listed patterns:

from session import run

# globstar lets ** cross folders; extglob adds patterns such as "anything but" and "one or more", which also work in
# ${var//pattern/replacement}. extglob must be on before Bash reads the line that uses it, so it gets a line of its own.
run("""
mkdir -p logs/api logs/web
touch logs/app.log logs/api/req.log logs/web/err.log draft-1.txt draft-2.txt final.txt notes.txt
count() { printf '%d:' "$#"; (( $# )) && printf ' [%s]' "$@"; printf '\\n'; }
count logs/*.log
shopt -s globstar
count **/*.log
!bash -c 'shopt -s extglob; echo !(draft-*).txt'
shopt -s extglob
count !(draft-*).txt
count +([a-z]).txt
touch photo.png scan.jpg icon.gif
count *.@(png|jpg)
f=report-v12-final-3.txt; printf '%s\\n' "${f//+([0-9])/N}"
""")

$ mkdir -p logs/api logs/web
$ touch logs/app.log logs/api/req.log logs/web/err.log draft-1.txt draft-2.txt final.txt notes.txt
$ count() { printf '%d:' "$#"; (( $# )) && printf ' [%s]' "$@"; printf '\n'; }
$ count logs/*.log
1: [logs/app.log]
$ shopt -s globstar
$ count **/*.log
3: [logs/api/req.log] [logs/app.log] [logs/web/err.log]
$ bash -c 'shopt -s extglob; echo !(draft-*).txt'
bash: -c: line 1: syntax error near unexpected token `('
bash: -c: line 1: `shopt -s extglob; echo !(draft-*).txt'
[exit status 2]
$ shopt -s extglob
$ count !(draft-*).txt
2: [final.txt] [notes.txt]
$ count +([a-z]).txt
2: [final.txt] [notes.txt]
$ touch photo.png scan.jpg icon.gif
$ count *.@(png|jpg)
2: [photo.png] [scan.jpg]
$ f=report-v12-final-3.txt; printf '%s\n' "${f//+([0-9])/N}"
report-vN-final-N.txt

logs/*.log found 1 file and **/*.log all 3. The manual describes ** as matching all files and zero or more folders, so a .log file at the top of the folder would have matched as well. !(draft-*).txt kept the two names that are not drafts, and +([a-z]).txt the names made only of letters. The last line used +([0-9]) inside ${f//pattern/N}, replacing each run of digits, because the same patterns work in parameter expansion and in [[ … ]].

The failed line in the session matters: extglob changes how Bash reads a line, so it must be on before the line that uses the pattern is read. Switched on in the same line, as in shopt -s extglob; echo !(draft-*).txt, the pattern is a syntax error. Put shopt -s extglob on a line of its own near the top of the script.

@(…) is the everyday one: *.@(png|jpg) matched the photo and the scan and left out the GIF, one pattern instead of two. Extended patterns can do much of what people reach for regular expressions to do, but they still match whole names, so +([a-z]).txt needed every character before .txt to be a letter.

GLOBSORT orders matches by size or time instead of name

Matches are normally sorted by name. Bash 5.3 adds the variable GLOBSORT, whose NEWS entry lists name, size, blocks, mtime, atime, ctime, numeric and none as the keys, in ascending or descending order. A leading - reverses the order:

from session import run

# Bash 5.3's GLOBSORT sorts the matches of a pattern by another key than the name; a leading - reverses the order.
# The three files are 5, 10 and 1 bytes long, so the name order and the two size orders all differ.
run("""
printf 'xxxxx' > apple.dat; printf 'xxxxxxxxxx' > kiwi.dat; printf 'x' > plum.dat
printf '[%s]\\n' *.dat
GLOBSORT=size; printf '[%s]\\n' *.dat
GLOBSORT=-size; printf '[%s]\\n' *.dat
unset GLOBSORT
""")

$ printf 'xxxxx' > apple.dat; printf 'xxxxxxxxxx' > kiwi.dat; printf 'x' > plum.dat
$ printf '[%s]\n' *.dat
[apple.dat]
[kiwi.dat]
[plum.dat]
$ GLOBSORT=size; printf '[%s]\n' *.dat
[plum.dat]
[apple.dat]
[kiwi.dat]
$ GLOBSORT=-size; printf '[%s]\n' *.dat
[kiwi.dat]
[apple.dat]
[plum.dat]
$ unset GLOBSORT

By name the order was apple, kiwi, plum; with GLOBSORT=size it became plum, apple, kiwi, 1, 5 and 10 bytes, and with -size the reverse. The manual lists the other keys: mtime, atime and ctime for the three file times, so -mtime puts the newest file first, blocks for the space on disk, and numeric, which sorts names made only of digits by their value, so 10 comes after 9. When two files tie on the key, the name decides. Older versions of Bash ignore the variable, so a script that relies on it should check BASH_VERSINFO first.

Practise: patterns against folders

The questions describe folders and ask which names a pattern matches, what an unmatched pattern does, and which option a script needs. The figure and the sessions hold every fact you need:

Interview questions

Warm-up (fresher to 2–5 years): what happens in Bash when a glob matches no files?

By default the pattern stays as the literal word, so ls *.md in a folder without Markdown files passes the text *.md to ls, which then reports that no such file exists. Two shell options change this. With nullglob the unmatched pattern expands to nothing at all, which suits loops that may have no work. With failglob it is an expansion error and the command does not run, which suits destructive commands such as rm. Whatever the setting, quote a pattern you mean literally, and check [ -e "$f" ] in a loop when you cannot change the options.

Two harder questions follow, about cleaning old logs safely and about a script that must also run on an older Bash. Make your plan, then open each first step:

Key takeaways

• *, ?, […] and [!…] match file names after word splitting, so a matched name with spaces stays one word, and a leading dot must be matched explicitly.
• A pattern that matches nothing stays as its own text unless nullglob removes it or failglob makes it an error, and globskipdots keeps . and .. out of matches since Bash 5.2.
• globstar makes ** search folders at any depth and extglob adds patterns such as !(…) and +(…), which must be switched on before the line that uses them is read.
• Bash 5.3’s GLOBSORT sorts matches by size, time or other keys, and older versions ignore it.