sort and uniq: counting and top-N reports

A busy day’s access log of 50,000 lines held the request /health on 12,418 of them and /api/orders on 7,342, and your boss wanted the top 10 paths by lunch. Scanning the file by eye would take the whole morning; sort access.log | uniq -c | sort -k1,1nr | head prints the same answer in a few seconds and in one line of shell.

You will put sort through its options for numbers, fields and reverse order, pair it with uniq to count duplicates, build a top-N report that always comes out in the same order, and know when LC_ALL=C makes the sort both faster and reproducible. The small text filters of the previous lesson are all the background you need.

Sort by number, by key, and in reverse

sort file reads the file into memory, puts the lines in order and prints them. The default order is by whole line, compared with the locale’s collation rules; with the session’s LC_ALL=C.UTF-8 it is byte by byte, so atta comes before rice and 10 comes before 2 because 1 is a smaller byte than 2. That is the trap learners meet first: a numeric column sorted as text puts 100 before 9.

"""sort's options for text: alphabetical, numeric, by field, reverse, unique and stable.

The session shows:
1. A plain sort compares strings byte by byte in LC_ALL=C, so "10" comes before "2".
2. -n reads each line as a number, so "2" comes before "10".
3. -k2,2 picks field 2 as the sort key (the second word on each line).
4. -t sets a different field separator for key-based sorting.
5. -r reverses the order; -u drops duplicates; -s keeps the input order for equal keys.
"""
from session import run

run("""
# Three product rows: name, quantity, price in paise
printf 'rice 10 7500\\nflour 2 4200\\natta 10 5100\\ndal 7 11000\\n' > stock.txt
cat stock.txt

# A plain sort: alphabetical by the whole line
sort stock.txt

# -n reads each line as a number: now 2 comes before 10
printf '10\\n2\\n100\\n7\\n' | sort
printf '10\\n2\\n100\\n7\\n' | sort -n

# -k2,2n: field 2 as a number (quantity). -k means "from this field"; ",2" stops it
sort -k2,2n stock.txt

# -k3,3nr: field 3 (price), largest first (-r)
sort -k3,3nr stock.txt

# -t sets the separator; -k1,1 sorts by the first comma-separated field
printf 'Pune,33.9\\nMumbai,33.5\\nDelhi,34.1\\n' | sort -t, -k1,1

# -u keeps one line per equal key: here one entry per quantity
sort -k2,2n -u stock.txt

# -s is stable: for equal keys, the original order is kept.
printf 'c 1\\na 2\\nb 1\\nd 2\\n' | sort -s -k2,2n
""")

# Three product rows: name, quantity, price in paise
$ printf 'rice 10 7500\nflour 2 4200\natta 10 5100\ndal 7 11000\n' > stock.txt
$ cat stock.txt
rice 10 7500
flour 2 4200
atta 10 5100
dal 7 11000
# A plain sort: alphabetical by the whole line
$ sort stock.txt
atta 10 5100
dal 7 11000
flour 2 4200
rice 10 7500
# -n reads each line as a number: now 2 comes before 10
$ printf '10\n2\n100\n7\n' | sort
10
100
2
7
$ printf '10\n2\n100\n7\n' | sort -n
2
7
10
100
# -k2,2n: field 2 as a number (quantity). -k means "from this field"; ",2" stops it
$ sort -k2,2n stock.txt
flour 2 4200
dal 7 11000
atta 10 5100
rice 10 7500
# -k3,3nr: field 3 (price), largest first (-r)
$ sort -k3,3nr stock.txt
dal 7 11000
rice 10 7500
atta 10 5100
flour 2 4200
# -t sets the separator; -k1,1 sorts by the first comma-separated field
$ printf 'Pune,33.9\nMumbai,33.5\nDelhi,34.1\n' | sort -t, -k1,1
Delhi,34.1
Mumbai,33.5
Pune,33.9
# -u keeps one line per equal key: here one entry per quantity
$ sort -k2,2n -u stock.txt
flour 2 4200
dal 7 11000
rice 10 7500
# -s is stable: for equal keys, the original order is kept.
$ printf 'c 1\na 2\nb 1\nd 2\n' | sort -s -k2,2n
c 1
b 1
a 2
d 2

The session built a small stock.txt with a name, a quantity and a price in paise on each line, then showed six options. -n reads each line as a number and gave 2, 7, 10, 100 for the four-number test; without it, sort ordered them 10, 100, 2, 7. -k2,2n sorts by field 2 as a number: the stock came out flour 2, dal 7, then the two tiedA line whose field at the chosen key compares equal to another line's. sort keeps them next to each other and, without -s, in an order the implementation does not promise. rows at quantity 10, in whatever order their input had. -k3,3nr sorted by price (field 3) with the largest first.

-t sets a different field separator. A CSV without -t, is one long field (every row sorts by its first byte, the same for every line); with -t, -k1,1 the session’s three city rows came out Delhi,Mumbai,Pune. Field numbering starts at 1; whitespace is the default separator and runs together, so a line with two spaces still has only one boundary.

Three one-letter flags change what counts: -r reverses the whole order, -u keeps the first line of each key run (so one line per unique key) and -s makes the sort stablestable sort: A sort that keeps the input order of lines whose keys compare equal; sort does not promise this by default, so pass --stable (or -s) when the order of ties matters., which keeps the input order of tied lines. The session’s sort -s -k2,2n for c 1, a 2, b 1, d 2 gave c 1, b 1, a 2, d 2: within each quantity, the names stayed in the order they came in. Without -s, the implementation may reorder them. Stability matters when you build a top-N on top: a stable sort with a tie-break key gives the same report every time.

After a sort, uniq counts duplicates

uniq collapses adjacent duplicate linesadjacent duplicates: Lines that are identical and sit next to each other, which uniq collapses into one. Lines whose duplicates are spread out survive uniq, so a sort before it is needed to group them. into one, so a file whose duplicates are spread out still shows them. A cities.txt of six lines with three Punes and two Delhis went through uniq unchanged in the session, because none of the duplicates sat next to each other. sort cities.txt | uniq dropped the file to three lines, one per city, because the sort put every copy of a name beside its siblings.

"""uniq counts adjacent duplicates, so sort first; then uniq -c, -d and -u give three different views.

The session shows:
1. uniq alone only drops ADJACENT duplicates: an unsorted file still has repeats.
2. sort piped into uniq keeps one line per value.
3. uniq -c prefixes each line with the count of its run.
4. uniq -d prints only lines that repeat; uniq -u prints only lines that do not.
5. uniq -f N skips N fields; uniq -w N compares the first N characters only.
"""
from session import run

run("""
# A log of visited cities (duplicates, not sorted)
printf 'Pune\\nDelhi\\nPune\\nMumbai\\nDelhi\\nPune\\n' > cities.txt
cat cities.txt

# uniq alone only drops ADJACENT repeats, so Pune and Delhi still appear twice
uniq cities.txt

# Sort first: all duplicates become adjacent, uniq keeps one of each
sort cities.txt | uniq

# uniq -c counts each run
sort cities.txt | uniq -c

# uniq -d prints only lines that repeat (shown once each)
sort cities.txt | uniq -d

# uniq -u prints only lines that do NOT repeat
sort cities.txt | uniq -u

# uniq -f 1 skips the first field, so a leading line number does not stop a run from merging
printf '1 cherry\\n2 cherry\\n3 banana\\n4 banana\\n5 banana\\n' | uniq -f 1 -c

# uniq -w 3 compares the first 3 characters only
printf 'india\\nindia-pale\\nindonesia\\n' | sort | uniq -w 3 -c
""")

# A log of visited cities (duplicates, not sorted)
$ printf 'Pune\nDelhi\nPune\nMumbai\nDelhi\nPune\n' > cities.txt
$ cat cities.txt
Pune
Delhi
Pune
Mumbai
Delhi
Pune
# uniq alone only drops ADJACENT repeats, so Pune and Delhi still appear twice
$ uniq cities.txt
Pune
Delhi
Pune
Mumbai
Delhi
Pune
# Sort first: all duplicates become adjacent, uniq keeps one of each
$ sort cities.txt | uniq
Delhi
Mumbai
Pune
# uniq -c counts each run
$ sort cities.txt | uniq -c
      2 Delhi
      1 Mumbai
      3 Pune
# uniq -d prints only lines that repeat (shown once each)
$ sort cities.txt | uniq -d
Delhi
Pune
# uniq -u prints only lines that do NOT repeat
$ sort cities.txt | uniq -u
Mumbai
# uniq -f 1 skips the first field, so a leading line number does not stop a run from merging
$ printf '1 cherry\n2 cherry\n3 banana\n4 banana\n5 banana\n' | uniq -f 1 -c
      2 1 cherry
      3 3 banana
# uniq -w 3 compares the first 3 characters only
$ printf 'india\nindia-pale\nindonesia\n' | sort | uniq -w 3 -c
      3 india

uniq -c prefixes each run with a left-padded count. For cities.txt, sort cities.txt | uniq -c printed 2 Delhi, 1 Mumbai and 3 Pune, with the counts in a column of their own. The count is the number of equal lines in the run, not the length of the line, and the output stays sorted by key (because sort was already in the pipe), not by count. The next section’s second sort is what puts the largest row at the top.

uniq -d keeps only lines that repeat; uniq -u keeps only lines that do not. On the sorted cities.txt the first printed Delhi and Pune, the second printed Mumbai: three ways of reading the same count. uniq -c -d together is often what a report wants: one row per repeated value, with its count.

Two options change what counts as equal. uniq -f N skips the first N whitespace-separated fields before comparing, so a leading line number or timestamp does not stop a run from merging; the session’s uniq -f 1 -c on five lines with line numbers in field 1 merged runs 1–2 (cherry) and 3–5 (banana) into 2 1 cherry and 3 3 banana. uniq -w N compares the first N characters only, so three words that share the first three bytes become one group of three.

Build a top-N report in one pipeline

Three stages produce the top-N report every busy log asks for. The first sort groups duplicates together, uniq -c counts each run, and a second sort on the first field as a number, with the largest first, puts the biggest rows at the top; head -n N keeps that many. Written in one line it is sort file | uniq -c | sort -k1,1nr -k2,2 | head -n N, four short filters sharing one pipelinepipeline: Commands joined by | or |&, each one's standard output flowing into the next one's standard input, all running at the same time. Its exit status is the last command's unless pipefail is set. More in “Pipelines, PIPESTATUS and pipefail”.

The second sort is where readers slip. sort -k1,1nr reads field 1 (the count uniq added) as a number and reverses, so 20 Pune comes before 10 Delhi because 20 is larger. Without -n, sort would compare the counts as text and place 1000 x after 10 x because 0 is a smaller byte than 1; and without -r, the smallest count would come first. -k2,2 is the tie-break by name: when two counts are equal, their rows come out in a fixed byte order of the second field, so the report never flips Pune and Delhi between runs.

"""The sort | uniq -c | sort pattern for a top-N report, with a deterministic tie-break by name.

The session shows:
1. The idiom: sort to group, uniq -c to count, sort -k1,1nr to put the largest counts first, head to pick N.
2. The tie-break -k2,2 breaks ties by name, so two cities with the same count always come out in the same order.
3. The resulting counts are left-padded; wrap with | awk '{ print $1, $2 }' when a program must read them.
4. A top-N over a two-column field (-k2) counts a column other than the whole line.
"""
from session import run

run("""
# A log of search hits by city; the top 3 should be the answer the report prints
printf 'Pune\\nDelhi\\nMumbai\\nPune\\nDelhi\\nBengaluru\\nPune\\nMumbai\\nDelhi\\nPune\\n' > hits.log

# One stage at a time: sort groups duplicates together
sort hits.log

# uniq -c then counts each group
sort hits.log | uniq -c

# The second sort re-orders by count, largest first (-k1,1nr), tie-break by name (-k2,2)
sort hits.log | uniq -c | sort -k1,1nr -k2,2

# head picks the top 3
sort hits.log | uniq -c | sort -k1,1nr -k2,2 | head -n 3

# A two-city tie: Pune and Delhi now both have 3; -k2,2 keeps Delhi before Pune every run
printf 'Pune\\nDelhi\\nMumbai\\nPune\\nDelhi\\nBengaluru\\nPune\\nMumbai\\nDelhi\\n' | sort | uniq -c | sort -k1,1nr -k2,2 | head -n 3

# Strip the left-padded count: awk prints two clean columns a program can read
sort hits.log | uniq -c | sort -k1,1nr -k2,2 | head -n 3 | awk '{ print $1, $2 }'

# Top-3 on a column other than the whole line: count unique words of a tab-separated log
printf 'Mon\\tPune\\nTue\\tDelhi\\nMon\\tPune\\nWed\\tPune\\nMon\\tDelhi\\n' > visits.tsv
cut -f 2 visits.tsv | sort | uniq -c | sort -k1,1nr -k2,2
""")

# A log of search hits by city; the top 3 should be the answer the report prints
$ printf 'Pune\nDelhi\nMumbai\nPune\nDelhi\nBengaluru\nPune\nMumbai\nDelhi\nPune\n' > hits.log
# One stage at a time: sort groups duplicates together
$ sort hits.log
Bengaluru
Delhi
Delhi
Delhi
Mumbai
Mumbai
Pune
Pune
Pune
Pune
# uniq -c then counts each group
$ sort hits.log | uniq -c
      1 Bengaluru
      3 Delhi
      2 Mumbai
      4 Pune
# The second sort re-orders by count, largest first (-k1,1nr), tie-break by name (-k2,2)
$ sort hits.log | uniq -c | sort -k1,1nr -k2,2
      4 Pune
      3 Delhi
      2 Mumbai
      1 Bengaluru
# head picks the top 3
$ sort hits.log | uniq -c | sort -k1,1nr -k2,2 | head -n 3
      4 Pune
      3 Delhi
      2 Mumbai
# A two-city tie: Pune and Delhi now both have 3; -k2,2 keeps Delhi before Pune every run
$ printf 'Pune\nDelhi\nMumbai\nPune\nDelhi\nBengaluru\nPune\nMumbai\nDelhi\n' | sort | uniq -c | sort -k1,1nr -k2,2 | head -n 3
      3 Delhi
      3 Pune
      2 Mumbai
# Strip the left-padded count: awk prints two clean columns a program can read
$ sort hits.log | uniq -c | sort -k1,1nr -k2,2 | head -n 3 | awk '{ print $1, $2 }'
4 Pune
3 Delhi
2 Mumbai
# Top-3 on a column other than the whole line: count unique words of a tab-separated log
$ printf 'Mon\tPune\nTue\tDelhi\nMon\tPune\nWed\tPune\nMon\tDelhi\n' > visits.tsv
$ cut -f 2 visits.tsv | sort | uniq -c | sort -k1,1nr -k2,2
      3 Pune
      2 Delhi

The session walked through the same hits.log of ten lines. sort hits.log gave the four cities grouped; sort hits.log | uniq -c printed 1 Bengaluru, 3 Delhi, 2 Mumbai, 4 Pune; the second sort re-ordered them to 4 Pune, 3 Delhi, 2 Mumbai, 1 Bengaluru; head -n 3 kept the first three. A second file where Delhi and Pune both reached a count of 3 showed the tie-break at work: -k2,2 put 3 Delhi ahead of 3 Pune because D is a smaller byte than P, and the same input gave the same order on every run.

The counts are left-padded (uutils used a column of width 6 in the session; GNU uses width 7), which is pleasant to read but awkward for a program that will parse the lines. Pipe through awk '{ print $1, $2 }', or tr -s ' ', to strip the padding to one space, as the session’s last stage did. For a top-N over a column other than the whole line, cut it out first: cut -f 2 visits.tsv | sort | uniq -c | sort -k1,1nr -k2,2 counted the top cities of a tab-separated log without looking at the day.

Locale, bytes and version order

A sort’s order depends on the locale. LC_ALL=C sort compares bytes, so uppercase letters come before lowercase ones (the ASCII block for A–Z sits before a–z) and the result is reproducible on any host. A dictionary locale such as en_US.UTF-8 interleaves cases and treats punctuation and spaces as if they were not there, which is friendly for a reader and surprising for a script. The POSIX.1-2024 sort page says the collation sequence comes from LC_COLLATE, so a server’s locale is as much a part of the sort’s order as the file.

"""Locale, bytes and version order: LC_ALL=C for byte-wise sorts, sort -V for versions and -h for sizes.

The session shows:
1. The locale changes the comparison: en_US.UTF-8 interleaves cases, LC_ALL=C puts upper case first.
2. LC_ALL=C is also faster for large files because sort compares bytes, not Unicode letters.
3. sort -V puts v1.2 before v1.9 before v1.10 (plain sort reads them as text).
4. sort -h compares human-readable sizes: 512 before 1K before 2M, as a disk-space report wants them.
5. The default locale leaves spaces and punctuation out when it compares, which can hide a leading space.
"""
from session import run

run("""
# Letters mixed with case: the default locale and a byte-wise sort disagree
printf 'B\\na\\nA\\nb\\n' > letters.txt

# LC_ALL=C compares byte values: upper case comes before lower case
LC_ALL=C sort letters.txt

# A dictionary locale interleaves the cases: a before A, b before B
LC_ALL=en_US.UTF-8 sort letters.txt

# -V reads release-style numbers correctly
printf 'v1.10\\nv1.2\\nv1.9\\n' | sort
printf 'v1.10\\nv1.2\\nv1.9\\n' | sort -V

# -h reads human-readable sizes
printf '2M\\n512\\n1K\\n100M\\n' | sort
printf '2M\\n512\\n1K\\n100M\\n' | sort -h

# -r reverses -h, so the largest row is first (useful for 'du -h | sort -hr | head')
printf '2M\\n512\\n1K\\n100M\\n' | sort -hr

# The default locale ignores leading spaces, so '  banana' lands between apple and cherry
printf 'apple\\n  banana\\ncherry\\n' | LC_ALL=en_US.UTF-8 sort
printf 'apple\\n  banana\\ncherry\\n' | LC_ALL=C sort
""")

# Letters mixed with case: the default locale and a byte-wise sort disagree
$ printf 'B\na\nA\nb\n' > letters.txt
# LC_ALL=C compares byte values: upper case comes before lower case
$ LC_ALL=C sort letters.txt
A
B
a
b
# A dictionary locale interleaves the cases: a before A, b before B
$ LC_ALL=en_US.UTF-8 sort letters.txt
a
A
b
B
# -V reads release-style numbers correctly
$ printf 'v1.10\nv1.2\nv1.9\n' | sort
v1.10
v1.2
v1.9
$ printf 'v1.10\nv1.2\nv1.9\n' | sort -V
v1.2
v1.9
v1.10
# -h reads human-readable sizes
$ printf '2M\n512\n1K\n100M\n' | sort
100M
1K
2M
512
$ printf '2M\n512\n1K\n100M\n' | sort -h
512
1K
2M
100M
# -r reverses -h, so the largest row is first (useful for 'du -h | sort -hr | head')
$ printf '2M\n512\n1K\n100M\n' | sort -hr
100M
2M
1K
512
# The default locale ignores leading spaces, so '  banana' lands between apple and cherry
$ printf 'apple\n  banana\ncherry\n' | LC_ALL=en_US.UTF-8 sort
apple
  banana
cherry
$ printf 'apple\n  banana\ncherry\n' | LC_ALL=C sort
  banana
apple
cherry

The session’s letters B, a, A, b sorted to A, B, a, b under LC_ALL=C and to a, A, b, B under LC_ALL=en_US.UTF-8. Point your script at LC_ALL=C when a report must agree between a developer’s laptop and the production server; the lesson’s own PATH does so, and the sorts are faster too, because the comparison is a byte compare instead of a Unicode collation lookup.

-V is the version-number sort GNU added for release tags: v1.2, v1.9, v1.10 land in that order, while a plain sort reads them as text and puts v1.10 before v1.2. -h is the human-readable size sort: 512, 1K, 2M, 100M, as a disk-usage report wants them. Pair -hr with du -h and head to find the biggest folders on a disk in one pipeline.

The default locale even ignores leading spaces when it compares. The session’s apple, banana (two spaces first) and cherry sorted to apple, banana, cherry under en_US.UTF-8 and to banana, apple, cherry under LC_ALL=C, because C compares the space byte first, and space is below a in ASCII. In a report that must be reproducible on every server, pick LC_ALL=C at the top of the script and move on; in a report shown to a reader of a given language, pick that reader’s locale and keep it on all the servers that serve them.

Practise

Build the top three paths of a toy access log in your own terminal. Save path.log with the ten lines /health, /health, /api/orders, /health, /, /health, /api/orders, /, /health, /api/orders. Then:

1. Run sort path.log | uniq -c and read the three counts off the output.
2. Add the second sort: sort path.log | uniq -c | sort -k1,1nr -k2,2, and check the row with the largest count sits on top.
3. Add head -n 3, then pipe into awk '{ print $1, $2 }' to see a clean two-column report.
4. Set FAIL_SORT=yes by hand: run the pipeline with the second sort as plain sort (no -k1,1nr). The order should change, and the row with ten hits should no longer be at the top; explain why from the byte-wise rule.
5. Change the file to add two paths with the same count. Confirm that two different runs with -k1,1nr -k2,2 give the same tie order; drop the -k2,2 and run twice to see that the tie order no longer agrees between runs.

The questions below use pipelines the lesson did not run. Work each one out before you open the explanation:

Interview questions

Warm-up: what does sort | uniq -c do, why does the sort come first, and how do you then order by count (fresher to 2–5 years)?

sort reads the file and puts the lines in order, which makes every copy of a line sit next to its siblings. uniq then collapses adjacent duplicates and -c prefixes each run with its count, so the result is one line per unique value with a count on the left. Without sort first, uniq only drops duplicates that happen to be adjacent, and a file with interleaved rows comes out almost unchanged. To order the counts largest first, a second sort -k1,1nr -k2,2 reads field 1 as a number and reverses, with a tie-break by field 2 for a deterministic order; head -n N keeps the top N.

The two harder questions ask about a daily report whose unique-visitor count drifted on busy days, and a top-N pipeline whose busiest row vanished from the top three once its count passed four digits. In each case, trace which stage held which bytes right before the symptom appeared, then compare your reasoning with the first step:

Key takeaways

• sort orders lines by whole-line bytes by default, so a numeric column needs -n and a non-default field key needs -k2,2 with the start and the end written out, and -t to set a separator other than whitespace.
• uniq only drops adjacent duplicates, so sort has to come first; uniq -c, uniq -d and uniq -u give three different views of the same groups: counts, repeating lines only, and non-repeating lines only.
• The top-N idiom is sort file | uniq -c | sort -k1,1nr -k2,2 | head -n N, with the second sort numeric, reversed and tie-broken by name so the report comes out the same way every time.
• LC_ALL=C makes sort compare bytes, which is predictable, reproducible and faster than a dictionary locale, while a reader-facing report keeps the reader’s locale on every server that serves it.
• sort -V reads version numbers like a human, so v1.2 lands before v1.10, and sort -h reads human-readable sizes, so 512 lands below 1K below 2M below 100M.