Text processing: the Unix toolbox interview questions

sort reads the file and puts the lines in order, which makes every copy of a line sit next to its siblings. uniq then collapses adjacent duplicates and -c prefixes each run with its count, so the result is one line per unique value with a count on the left. Without sort first, uniq only drops duplicates that happen to be adjacent, and a file with interleaved rows comes out almost unchanged. To order the counts largest first, a second sort -k1,1nr -k2,2 reads field 1 as a number and reverses, with a tie-break by field 2 for a deterministic order; head -n N keeps the top N.

Three that come up every day: -i ignores case, so a search for error picks error, Error and ERROR without listing every spelling; -r reads every file under a folder and tags each match with the file's path, which turns grep into a quick "find in files"; and -q prints nothing and sets the exit status, which is why if grep -q PATTERN FILE; then …; fi is the usual test for "the file contains PATTERN". After those, -n for line numbers in a long file, -A or -B to see what came before or after a match, and -F to tell grep the pattern is plain text are the ones most scripts need. Each follows the GNU grep manual and POSIX.1-2024, so a script that uses them works on dash and bash alike.

POSIX defines two dialects, basic regular expressions (BRE) and extended regular expressions (ERE), with the same engine and different metacharacters. BRE is the default of grep and of sed; ERE is what grep -E, sed -E, awk and Bash's [[ string =~ regex ]] read. In BRE the characters + ? | ( ) { are literal and need a backslash to become metacharacters, in ERE they are metacharacters and a backslash makes them literal. POSIX regex has no \d, \s or lookaround; GNU grep accepts \w and \s as extensions, but a script that must run on BSD tools writes [[:digit:]] and [[:space:]] instead. The leftmost-longest rule applies to both dialects.

Both are the substitute command, and both read the current line into the pattern space, scan it from the left and try to replace old with new. Without the g flag, s/// stops after the first match on the line, so sed 's/cat/dog/' on 'ERROR cat sat on cat' printed 'ERROR dog sat on cat', with the second cat left alone. With g, sed keeps scanning after each replacement and changes every match on the line, so the same input became 'ERROR dog sat on dog'. A number flag N, such as s/cat/dog/2, picks only the Nth match and stops; p prints the pattern space when s/// matched (useful with -n); and i makes the match case-insensitive, which POSIX.1-2024 added to standard sed and which BSD and GNU sed both accept.

Three of awk's built-in names a learner meets first. NR is the record number across all the input awk reads, counting from 1. After awk has read the six lines of people.csv, NR is 6; after a second file, NR keeps going. NF is the number of fields in the CURRENT record, after awk has split it on FS, so NF is 3 for every row of a comma-separated people.csv read with -F,, and 1 if the same file is read with the default white-space FS. $NF is the LAST field of the current record: the field whose index is the VALUE of NF, so on Asha,34,Pune with -F, it is the string Pune. NF is a count; $NF is a value. A sister name, FNR, is NR reset at the start of each file.

Pipe find's -print0 to a reader that splits on NUL instead of whitespace. The two readers that fit are xargs -0 and a Bash loop with while IFS= read -r -d '' name. The nul session's run on a tree with a tab-in-name file and a newline-in-name file showed every name arriving whole through -print0 | xargs -0, while the plain find | xargs form split the same four names into eight argument words. -print0 ends each path with a NUL byte, which no file name can hold, so the next tool has an unambiguous separator; POSIX.1-2024 added both -print0 and -0 to the standard. The alternative is find . -type f -exec cmd {} +, which fires far fewer processes than -exec cmd {} \; and hands each name over whole, but is less flexible when the command needs the path somewhere other than the end of its line.

. is the identity filter: it emits its input unchanged, so jq '.' orders.json pretty-prints the whole object. .[] is the array or object iterator: it emits each element of an array (or each value of an object) as its own value, so the next stage of a filter sees one element at a time instead of the whole collection. Reach for . when you only want jq to validate and format, or at the start of a filter where the input flows in as is. Reach for .[] whenever the next stage is "do something per item": .orders[] | select(...), .orders[] | .id, or [.orders[] | .amount] | add to sum. The two forms combine: .orders[] is read as .orders | .[], and the jq manual's "Array/Object Value Iterator" entry gives both forms.

cut -d, splits each line on every comma without understanding CSV quoting. When a field contains a comma inside double quotes, like "Mehta, Ravi", cut sees two fields instead of one. The same happens with awk -F, and IFS=, read. RFC 4180 defines quoting, but these tools do not implement it. gawk --csv (gawk 5.3+) parses quoted fields correctly, keeping the comma inside its field and un-escaping doubled quotes.

A UTC epoch names one instant in time; a UTC ISO 8601 string with a Z suffix says the same instant with the fields spelt out. Both are unambiguous because UTC has no daylight saving: the mapping to any local wall clock is a pure function of the tz database. A local time stored without a zone leaves the reader to guess, and even a local time stored with a zone goes through a DST-aware mapping that is not one-to-one in one hour each spring and autumn. Storing UTC avoids both problems: the field is unambiguous, every tool reads the same instant (jq, awk, GNU date, BSD date), and converting for a user is one strftime at the edge. The rule works the same for logs, queues, databases and schedulers, which is why it survives a move between machines or regions.