Bash & the POSIX Shell Module 11 – Text processing: the Unix toolbox

awk programs: arrays, joins and reports

Group data with awk associative arrays, join two files with the FNR==NR idiom, format reports with printf and write multi-line programs with functions.

Advanced
10 minutes
Examples run with Bash 5.3.20, dash 0.5.13.5, shellcheck 0.11.0 and Python 3.14.8
By MySmartCoPilot

Free: the whole lesson. With a pass: the model answers to its 2 interview questions.

What you will learn

Group values by key with associative arrays and handle the unspecified for-in order
Join two files with the FNR==NR idiom and detect missing or duplicate keys
Format tabular reports with printf and calculate totals and percentages
Write output to files and commands and use close before reading back
Create maintainable awk programs in files with comments and functions

Before you start

awk: fields, patterns and actions

What you’ll learn · Before you start

What you will learn

Group values by key with associative arrays and handle the unspecified for-in order
Join two files with the FNR==NR idiom and detect missing or duplicate keys
Format tabular reports with printf and calculate totals and percentages
Write output to files and commands and use close before reading back
Create maintainable awk programs in files with comments and functions

Before you start

awk: fields, patterns and actions

On this page

A kirana shop has 12 items across three categories. One awk program, run once, groups the sales by category, computes each category’s share of the grand total, finds the top seller and prints a formatted report with printf. The shell alternative would need several pipelines, temporary files and a loop to do the arithmetic.

You will group data by key with associative arrays, detect the silent traps of the FNR==NR join idiom, format output with printf, send output to files and commands, and write multi-line awk programs with comments and functions. The previous lesson covers fields, patterns and actions; this one builds on them.

Group by key with associative arrays

An An awk array that stores values keyed by strings, accessed with array[key], created on first use. uses strings as keys, not numbers. Writing count[$2]++ creates an entry keyed by the second field and increments it; sum[$2] += $3 accumulates a running total. Neither operation needs a declaration: the array and its entry appear on first use, starting at zero for numbers and the empty string for text.

Counting and summing by category Python · arrays_session.py

"""awk associative arrays: count, sum, for-in order, sorting the output.

The session shows:
1. count[$2]++ groups sales by category
2. sum[$2]+=$3 keeps running totals
3. for (k in count) visits keys in an unspecified order (different between awk versions)
4. Piping awk output to sort gives a deterministic order
5. gawk's PROCINFO["sorted_in"] sorts keys inside the for-in loop
"""
from session import run, write

write("sales.csv", """\
Milk,Dairy,45
Bread,Bakery,30
Cheese,Dairy,120
Paneer,Dairy,80
Rusk,Bakery,25
Butter,Dairy,55
Bun,Bakery,15
Ghee,Dairy,250
""")

run("""
cat sales.csv

# Count items per category with count[$2]++
awk -F, '{ count[$2]++ } END { for (k in count) print k, count[k] }' sales.csv

# The for-in order is unspecified: gawk may print differently than mawk
# Verify with /usr/bin/awk (macOS awk, not gawk)
/usr/bin/awk -F, '{ count[$2]++ } END { for (k in count) print k, count[k] }' sales.csv

# Sum amounts per category with sum[$2]+=$3
awk -F, '{ sum[$2]+=$3 } END { for (k in sum) print k, sum[k] }' sales.csv

# Combine count and sum in one pass
awk -F, '{ count[$2]++; sum[$2]+=$3 } END { for (k in count) print k, count[k], sum[k] }' sales.csv

# Pipe to sort for a deterministic order (the portable fix)
awk -F, '{ count[$2]++; sum[$2]+=$3 } END { for (k in count) print k, count[k], sum[k] }' sales.csv | sort

# gawk's PROCINFO["sorted_in"] sorts inside the for-in loop itself
awk -F, '{ count[$2]++; sum[$2]+=$3 } END { PROCINFO["sorted_in"]="@ind_str_asc"; for (k in count) print k, count[k], sum[k] }' sales.csv

# Check if a key exists with (key in array)
awk -F, '{ count[$2]++ } END { if ("Dairy" in count) print "Dairy:", count["Dairy"]; if ("Frozen" in count) print "Frozen found" }' sales.csv

# Delete an entry with delete array[key]
awk -F, '{ count[$2]++ } END { delete count["Bakery"]; for (k in count) print k, count[k] }' sales.csv
""")

Output

$ cat sales.csv
Milk,Dairy,45
Bread,Bakery,30
Cheese,Dairy,120
Paneer,Dairy,80
Rusk,Bakery,25
Butter,Dairy,55
Bun,Bakery,15
Ghee,Dairy,250
# Count items per category with count[$2]++
$ awk -F, '{ count[$2]++ } END { for (k in count) print k, count[k] }' sales.csv
Dairy 5
Bakery 3
# The for-in order is unspecified: gawk may print differently than mawk
# Verify with /usr/bin/awk (macOS awk, not gawk)
$ /usr/bin/awk -F, '{ count[$2]++ } END { for (k in count) print k, count[k] }' sales.csv
Bakery 3
Dairy 5
# Sum amounts per category with sum[$2]+=$3
$ awk -F, '{ sum[$2]+=$3 } END { for (k in sum) print k, sum[k] }' sales.csv
Dairy 550
Bakery 70
# Combine count and sum in one pass
$ awk -F, '{ count[$2]++; sum[$2]+=$3 } END { for (k in count) print k, count[k], sum[k] }' sales.csv
Dairy 5 550
Bakery 3 70
# Pipe to sort for a deterministic order (the portable fix)
$ awk -F, '{ count[$2]++; sum[$2]+=$3 } END { for (k in count) print k, count[k], sum[k] }' sales.csv | sort
Bakery 3 70
Dairy 5 550
# gawk's PROCINFO["sorted_in"] sorts inside the for-in loop itself
$ awk -F, '{ count[$2]++; sum[$2]+=$3 } END { PROCINFO["sorted_in"]="@ind_str_asc"; for (k in count) print k, count[k], sum[k] }' sales.csv
Bakery 3 70
Dairy 5 550
# Check if a key exists with (key in array)
$ awk -F, '{ count[$2]++ } END { if ("Dairy" in count) print "Dairy:", count["Dairy"]; if ("Frozen" in count) print "Frozen found" }' sales.csv
Dairy: 5
# Delete an entry with delete array[key]
$ awk -F, '{ count[$2]++ } END { delete count["Bakery"]; for (k in count) print k, count[k] }' sales.csv
Dairy 5

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 arrays_session.py

The session shows 8 items across 2 categories. count[$2]++ counted 5 Dairy and 3 Bakery; sum[$2] += $3 totalled 550 and 70 rupees. The one-pass version { count[$2]++; sum[$2]+=$3 } does both in a single scan of the file, which matters on large logs.

The for (k in count) loop visits every key, but the order is unspecified. gawk printed Dairy first, then Bakery; macOS awk (run as /usr/bin/awk) printed Bakery first. A script that expects a fixed order breaks when it moves to another machine.

Portable sorted output

Pipe the awk output to sort for a deterministic order on every awk. gawk’s PROCINFO["sorted_in"] = "@ind_str_asc" sorts inside the loop itself, which is faster but gawk-only.

Two more operations complete the array toolbox: (key in array) tests whether a key exists, and delete array[key] removes it. The session’s ("Dairy" in count) returned true, ("Frozen" in count) returned false, and delete count["Bakery"] left only Dairy in the output.

Checkpoint

Check: Associative arrays and for-in order

Question 1 of 2 On a CSV with fields item,category,price, what does awk -F, '{ sum[$2]+=$3 } END { print sum["Dairy"] }' print?
Choose one answer.

An error, because sum was not declared The last price of a Dairy item The total of prices where field 2 is Dairy The count of Dairy rows

Show the answer: Question 1 of 2

Answer: The total of prices where field 2 is Dairy

The += operator adds each price to the running total keyed by category. The END block then prints the sum for one key. The arrays session showed this pattern with 5 Dairy items summing to 550.
Question 2 of 2 A script prints for (k in count) print k and expects categories in alphabetical order. On gawk it works; on macOS awk the order differs. What is the portable fix?
Choose one answer.

Declare the array with a sorted keyword Pipe the output to sort Add PROCINFO["sorted_in"] before the loop Use for (k = 1; k <= length(count); k++)

Show the answer: Question 2 of 2

Answer: Pipe the output to sort

The POSIX standard leaves for-in order unspecified. The arrays session showed gawk and macOS awk printing Dairy and Bakery in opposite orders; piping to sort fixed it.

A formatted report with printf

A grouped report needs column alignment, headers and computed values. awk’s printf is the C-style formatter: printf "%-10s %5d", name, qty prints a left-aligned string of at most 10 characters and a right-aligned integer of at least 5 digits. Percentages use %5.1f%% (5 characters, 1 decimal, literal %).

A kirana-shop sales report Python · report_session.py

"""awk report: a kirana-shop sales summary with totals, averages, top category and percentage share.

The session shows:
1. A multi-line awk program (via awk -f) that groups sales by category
2. printf for formatted output with column alignment
3. Calculating the top category and percentage share in END
"""
from session import run, write

write("kirana.csv", """\
Milk,Dairy,45
Bread,Bakery,30
Cheese,Dairy,120
Paneer,Dairy,80
Rice,Staples,65
Rusk,Bakery,25
Butter,Dairy,55
Atta,Staples,48
Bun,Bakery,15
Ghee,Dairy,250
Dal,Staples,72
Sugar,Staples,40
""")

write("report.awk", """\
BEGIN { FS = "," }
{ count[$2]++; sum[$2] += $3; grand += $3 }
END {
    printf "%-10s %5s %8s %6s\\n", "Category", "Items", "Total", "Share"
    printf "%-10s %5s %8s %6s\\n", "--------", "-----", "-----", "-----"
    PROCINFO["sorted_in"] = "@ind_str_asc"
    for (k in sum) {
        pct = (sum[k] / grand) * 100
        printf "%-10s %5d %8d %5.1f%%\\n", k, count[k], sum[k], pct
    }
    printf "%-10s %5s %8s %6s\\n", "--------", "-----", "-----", "-----"
    printf "%-10s %5d %8d\\n", "TOTAL", NR, grand
}
""")

write("topcategory.awk", """\
BEGIN { FS = "," }
{ sum[$2] += $3 }
END {
    max_cat = ""; max_val = 0
    for (k in sum) { if (sum[k] > max_val) { max_val = sum[k]; max_cat = k } }
    print "Top category:", max_cat, "with", max_val, "rupees"
}
""")

write("avgstats.awk", """\
BEGIN { FS = "," }
{ count[$2]++; sum[$2] += $3 }
END {
    PROCINFO["sorted_in"] = "@ind_str_asc"
    printf "%-10s %5s %8s %7s\\n", "Category", "Items", "Total", "Average"
    printf "%-10s %5s %8s %7s\\n", "--------", "-----", "-----", "-------"
    for (k in sum) {
        avg = sum[k] / count[k]
        printf "%-10s %5d %8d %7.1f\\n", k, count[k], sum[k], avg
    }
}
""")

run("""
cat kirana.csv

# A one-line summary: count and sum per category (unsorted)
awk -F, '{ count[$2]++; sum[$2]+=$3 } END { for (k in count) print k, count[k], sum[k] }' kirana.csv

# The formatted report using awk -f
cat report.awk
awk -f report.awk kirana.csv

# Find the top category by total sales
cat topcategory.awk
awk -f topcategory.awk kirana.csv

# Report with averages using PROCINFO for sorted output
cat avgstats.awk
awk -f avgstats.awk kirana.csv
""")

Output

$ cat kirana.csv
Milk,Dairy,45
Bread,Bakery,30
Cheese,Dairy,120
Paneer,Dairy,80
Rice,Staples,65
Rusk,Bakery,25
Butter,Dairy,55
Atta,Staples,48
Bun,Bakery,15
Ghee,Dairy,250
Dal,Staples,72
Sugar,Staples,40
# A one-line summary: count and sum per category (unsorted)
$ awk -F, '{ count[$2]++; sum[$2]+=$3 } END { for (k in count) print k, count[k], sum[k] }' kirana.csv
Staples 4 225
Dairy 5 550
Bakery 3 70
# The formatted report using awk -f
$ cat report.awk
BEGIN { FS = "," }
{ count[$2]++; sum[$2] += $3; grand += $3 }
END {
    printf "%-10s %5s %8s %6s\n", "Category", "Items", "Total", "Share"
    printf "%-10s %5s %8s %6s\n", "--------", "-----", "-----", "-----"
    PROCINFO["sorted_in"] = "@ind_str_asc"
    for (k in sum) {
        pct = (sum[k] / grand) * 100
        printf "%-10s %5d %8d %5.1f%%\n", k, count[k], sum[k], pct
    }
    printf "%-10s %5s %8s %6s\n", "--------", "-----", "-----", "-----"
    printf "%-10s %5d %8d\n", "TOTAL", NR, grand
}
$ awk -f report.awk kirana.csv
Category   Items    Total  Share
--------   -----    -----  -----
Bakery         3       70   8.3%
Dairy          5      550  65.1%
Staples        4      225  26.6%
--------   -----    -----  -----
TOTAL         12      845
# Find the top category by total sales
$ cat topcategory.awk
BEGIN { FS = "," }
{ sum[$2] += $3 }
END {
    max_cat = ""; max_val = 0
    for (k in sum) { if (sum[k] > max_val) { max_val = sum[k]; max_cat = k } }
    print "Top category:", max_cat, "with", max_val, "rupees"
}
$ awk -f topcategory.awk kirana.csv
Top category: Dairy with 550 rupees
# Report with averages using PROCINFO for sorted output
$ cat avgstats.awk
BEGIN { FS = "," }
{ count[$2]++; sum[$2] += $3 }
END {
    PROCINFO["sorted_in"] = "@ind_str_asc"
    printf "%-10s %5s %8s %7s\n", "Category", "Items", "Total", "Average"
    printf "%-10s %5s %8s %7s\n", "--------", "-----", "-----", "-------"
    for (k in sum) {
        avg = sum[k] / count[k]
        printf "%-10s %5d %8d %7.1f\n", k, count[k], sum[k], avg
    }
}
$ awk -f avgstats.awk kirana.csv
Category   Items    Total Average
--------   -----    ----- -------
Bakery         3       70    23.3
Dairy          5      550   110.0
Staples        4      225    56.2

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 report_session.py

The report program runs through three stages. During the main loop, { count[$2]++; sum[$2] += $3; grand += $3 } builds two arrays and a grand total. In the END block, the program prints headers with printf, iterates the categories in sorted order (via PROCINFO["sorted_in"]) and computes each category’s share as (sum[k] / grand) * 100. The result is a table that shows Dairy at 65.1 %, Bakery at 8.3 % and Staples at 26.6 %, then a footer with the grand total of 845 rupees across 12 items.

Finding the top category is a loop inside END: start with max_val = 0, compare each sum[k] and keep the winner. The session printed “Top category: Dairy with 550 rupees”.

Key idea

Arrays plus END let awk hold state across all lines, so one pass computes totals, averages, maximums and percentages without a second scan or a temporary file.

Checkpoint

Check: Formatted reports with printf

Question 1 of 2 Which printf format prints a percentage with one decimal place and a literal percent sign?
Choose one answer.

"%5.1d%%" "%5.1f\\%" "%5.1f%" "%5.1f%%"

Show the answer: Question 1 of 2

Answer: "%5.1f%%"

The report session used printf "...%5.1f%%..." to print 65.1% for Dairy. The double percent is the awk (and C) way to write a literal percent sign.
Question 2 of 2 In an END block, you want to find the key with the highest sum. Which approach works?
Choose one answer.

Call max(sum) to get the highest value Loop over keys, compare each sum[k] to max_val, update max_val and max_cat Use PROCINFO["sorted_in"] = "@val_num_desc" and take the first key Use sort_by(sum) and take the last element

Show the answer: Question 2 of 2

Answer: Loop over keys, compare each sum[k] to max_val, update max_val and max_cat

The report session showed the loop: max_val = 0; for (k in sum) if (sum[k] > max_val) { max_val = sum[k]; max_cat = k }.

Join two files with FNR and NR

A common task is adding a lookup field: given a list of orders and a list of customers, print each order with its customer name. The The idiom FNR==NR { load; next } { use } in awk: the first file is loaded into an array, the second file is looked up against it. handles this in one awk call.

FNR is the record number within the current file; NR is the global record number. On the first file they are equal, so FNR==NR { name[$1] = $2; next } loads each customer into an array keyed by id and skips to the next line. On the second file FNR resets to 1 while NR keeps counting, so the condition is false and the second block runs, printing each order with its looked-up name.

The flow below follows awk through both files on the lesson’s own data: the four customers load into name[], then each order is looked up against it, and order O105 ends with a blank because its id was never loaded.

Interactive · stage by stage How FNR==NR joins two files Follow awk through one join. The first file loads into an array keyed by id; the second file is looked up against it, and a row whose id was never loaded prints a blank.

The drawing shows the whole scene; its 6 steps are listed below.

Every step as text

Open the first file. awk opens customers.csv first. FNR counts records within the current file and NR counts across all files, so on the first file they are always equal and FNR==NR is true.
Load a customer. The record C001,Asha Patel has FNR and NR both 1, so FNR==NR is true. The block name[$1]=$2 stores name[C001] as Asha Patel, and next skips straight to the following record.
Finish loading file 1. awk repeats for all four customers, so name[] holds C001 to C004. C005 is never in customers.csv, so it is never loaded: that gap is what makes order O105 print a blank later.
Switch to the second file. When awk opens orders.csv, FNR resets to 1 but NR keeps counting (5, 6, ...). Now FNR==NR is false, so the load block is skipped and the lookup block runs for every order.
Look up a matched order. Order O101,C001,450 looks up name[C001], which is Asha Patel, so it prints the order line with Asha Patel appended. Orders O102 to O104 match the same way.
The missing-key trap. Order O105,C005,500 looks up name[C005], which was never loaded, so awk returns the empty string and prints the order with a blank name. No error is raised; guard with ($2 in name).

Joining customers to orders Python · join_session.py

"""awk join: the FNR==NR idiom for joining two files, and what goes wrong.

The session shows:
1. FNR==NR loads the first file into an array, then looks up the second file
2. Missing keys silently drop data (empty string)
3. Duplicate keys silently overwrite (last one wins)
4. A safer join that reports misses
"""
from session import run, write

write("customers.csv", """\
C001,Asha Patel
C002,Ravi Kumar
C003,Meera Iyer
C004,Kiran Shah
""")

write("orders.csv", """\
O101,C001,450
O102,C002,320
O103,C001,180
O104,C003,275
O105,C005,500
""")

run("""
cat customers.csv
cat orders.csv

# Join: add customer names to orders using FNR==NR (single line)
awk -F, 'FNR==NR { name[$1]=$2; next } { print $0, name[$2] }' customers.csv orders.csv

# What happened to order O105? Its customer C005 is missing, so name["C005"] is empty
# The join silently prints the order with a blank name field

# Duplicate keys: if a customer appears twice, the second name wins
printf 'C001,Asha Patel\\nC001,Asha P.\\n' > dup_customers.csv
cat dup_customers.csv
awk -F, 'FNR==NR { name[$1]=$2; next } { print $0, name[$2] }' dup_customers.csv orders.csv

# Check if a key exists before printing with (key in array)
awk -F, 'FNR==NR { name[$1]=$2; next } ($2 in name) { print $0, name[$2] }' customers.csv orders.csv

# Report missing keys to stderr
awk -F, 'FNR==NR { name[$1]=$2; next } { if ($2 in name) print $0, name[$2]; else print $0, "(unknown)" > "/dev/stderr" }' customers.csv orders.csv

# Use OFS to set the output field separator for comma-separated output
awk -F, -v OFS=, 'FNR==NR { name[$1]=$2; next } ($2 in name) { print $1, $2, $3, name[$2] }' customers.csv orders.csv

rm dup_customers.csv
""")

Output

$ cat customers.csv
C001,Asha Patel
C002,Ravi Kumar
C003,Meera Iyer
C004,Kiran Shah
$ cat orders.csv
O101,C001,450
O102,C002,320
O103,C001,180
O104,C003,275
O105,C005,500
# Join: add customer names to orders using FNR==NR (single line)
$ awk -F, 'FNR==NR { name[$1]=$2; next } { print $0, name[$2] }' customers.csv orders.csv
O101,C001,450 Asha Patel
O102,C002,320 Ravi Kumar
O103,C001,180 Asha Patel
O104,C003,275 Meera Iyer
O105,C005,500
# What happened to order O105? Its customer C005 is missing, so name["C005"] is empty
# The join silently prints the order with a blank name field
# Duplicate keys: if a customer appears twice, the second name wins
$ printf 'C001,Asha Patel\nC001,Asha P.\n' > dup_customers.csv
$ cat dup_customers.csv
C001,Asha Patel
C001,Asha P.
$ awk -F, 'FNR==NR { name[$1]=$2; next } { print $0, name[$2] }' dup_customers.csv orders.csv
O101,C001,450 Asha P.
O102,C002,320
O103,C001,180 Asha P.
O104,C003,275
O105,C005,500
# Check if a key exists before printing with (key in array)
$ awk -F, 'FNR==NR { name[$1]=$2; next } ($2 in name) { print $0, name[$2] }' customers.csv orders.csv
O101,C001,450 Asha Patel
O102,C002,320 Ravi Kumar
O103,C001,180 Asha Patel
O104,C003,275 Meera Iyer
# Report missing keys to stderr
$ awk -F, 'FNR==NR { name[$1]=$2; next } { if ($2 in name) print $0, name[$2]; else print $0, "(unknown)" > "/dev/stderr" }' customers.csv orders.csv
O105,C005,500 (unknown)
O101,C001,450 Asha Patel
O102,C002,320 Ravi Kumar
O103,C001,180 Asha Patel
O104,C003,275 Meera Iyer
# Use OFS to set the output field separator for comma-separated output
$ awk -F, -v OFS=, 'FNR==NR { name[$1]=$2; next } ($2 in name) { print $1, $2, $3, name[$2] }' customers.csv orders.csv
O101,C001,450,Asha Patel
O102,C002,320,Ravi Kumar
O103,C001,180,Asha Patel
O104,C003,275,Meera Iyer
$ rm dup_customers.csv

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 join_session.py

Four orders printed their customer names; order O105 printed a blank because C005 is not in the customer file. awk never warns about a missing key: name["C005"] simply returns the empty string.

Two traps hide in this pattern. Missing keys silently drop data, as O105 did. Duplicate keys overwrite: when a customer appeared twice, the second row won, so “Asha Patel” became “Asha P.” for every order. The fix is ($2 in name) as a pattern before the action, which skips rows whose key is absent, or an if that prints to stderr when the key is missing.

A join that silently loses rows

Wrong: awk 'FNR==NR { a[$1]; next } { print $0, a[$2] }' keys vals prints every line of vals, even those with no match (their lookup is empty). Right: add ($2 in a) as a pattern, { if ($2 in a) ... } as a test, or print missing keys to stderr so you know they were dropped.

Use -v OFS=, when you need the output fields separated by commas. The session’s final command printed O101,C001,450,Asha Patel as four comma-separated values.

Checkpoint

Check: FNR==NR join idiom

Question 1 of 2 In awk, what does FNR==NR test?
Choose one answer.

That the current field equals the record number That the current line is from the first file That the record is not a header That the file has not changed since the last line

Show the answer: Question 1 of 2

Answer: That the current line is from the first file

The join session used FNR==NR { name[$1]=$2; next } to load the first file. On the second file, FNR reset to 1 while NR continued, so the condition was false and the lookup block ran.
Question 2 of 2 A join with FNR==NR prints orders with customer names. Order O105 has customer id C005, which is not in the customer file. What does the join print for O105?
Choose one answer.

The literal string (unknown) Nothing, because the order is skipped An error about a missing key The order line with a blank where the name should be

Show the answer: Question 2 of 2

Answer: The order line with a blank where the name should be

The join session showed O105 printed with a blank at the end. The fix is ($2 in name) as a pattern or an if block that sends missing keys to stderr.

Output to files and commands

Inside an awk program, print $1 > "file" writes to a file. The file is truncated once per awk run and appended to for every subsequent print in that run. print $1 >> "file" always appends. print $1 | "cmd" pipes to a command; each unique command string opens one pipe.

Writing to files and piping to commands Python · output_session.py

"""awk output: print to files and commands, and close() before reading back.

The session shows:
1. print > "file" truncates once per run, then appends within the run
2. print >> "file" always appends
3. print | "cmd" pipes to a command
4. close("file") before reading the file back in the same run
"""
from session import run, write

write("items.csv", """\
Milk,Dairy
Bread,Bakery
Cheese,Dairy
Rice,Staples
Paneer,Dairy
Rusk,Bakery
Atta,Staples
""")

write("split.awk", """\
BEGIN { FS = "," }
{ print $1 > ($2 ".txt") }
""")

write("readback_close.awk", """\
BEGIN { FS = "," }
{ print $1 > "sorted_items.txt" }
END {
    close("sorted_items.txt")
    while ((getline line < "sorted_items.txt") > 0)
        print "Read back:", line
}
""")

write("readback_noclose.awk", """\
BEGIN { FS = "," }
{ print $1 > "sorted_items.txt" }
END {
    while ((getline line < "sorted_items.txt") > 0)
        print "Read back:", line
    print "Lines read: none visible, file not fully flushed"
}
""")

run("""
cat items.csv

# print > "file" truncates once, then appends within the same awk run
awk -f split.awk items.csv
cat Dairy.txt
cat Bakery.txt
cat Staples.txt

# Running awk again truncates the files (not appending to old content)
awk -f split.awk items.csv
cat Dairy.txt

# print >> "file" always appends
echo "Biscuit" >> Bakery.txt
awk -F, '{ print $1 >> "Bakery.txt" }' items.csv
cat Bakery.txt

# Clean up for the next demo
rm -f Dairy.txt Bakery.txt Staples.txt

# print | "cmd" pipes output to a command
awk -F, '{ print $1 | "sort" }' items.csv

# close() before reading a file you wrote in the same run
cat readback_close.awk
awk -f readback_close.awk items.csv

# Without close(), the file may not be visible to getline yet
rm -f sorted_items.txt
cat readback_noclose.awk
awk -f readback_noclose.awk items.csv
# The file exists after awk finishes, even though getline saw nothing
cat sorted_items.txt
rm -f sorted_items.txt
""")

Output

$ cat items.csv
Milk,Dairy
Bread,Bakery
Cheese,Dairy
Rice,Staples
Paneer,Dairy
Rusk,Bakery
Atta,Staples
# print > "file" truncates once, then appends within the same awk run
$ awk -f split.awk items.csv
$ cat Dairy.txt
Milk
Cheese
Paneer
$ cat Bakery.txt
Bread
Rusk
$ cat Staples.txt
Rice
Atta
# Running awk again truncates the files (not appending to old content)
$ awk -f split.awk items.csv
$ cat Dairy.txt
Milk
Cheese
Paneer
# print >> "file" always appends
$ echo "Biscuit" >> Bakery.txt
$ awk -F, '{ print $1 >> "Bakery.txt" }' items.csv
$ cat Bakery.txt
Bread
Rusk
Biscuit
Milk
Bread
Cheese
Rice
Paneer
Rusk
Atta
# Clean up for the next demo
$ rm -f Dairy.txt Bakery.txt Staples.txt
# print | "cmd" pipes output to a command
$ awk -F, '{ print $1 | "sort" }' items.csv
Atta
Bread
Cheese
Milk
Paneer
Rice
Rusk
# close() before reading a file you wrote in the same run
$ cat readback_close.awk
BEGIN { FS = "," }
{ print $1 > "sorted_items.txt" }
END {
    close("sorted_items.txt")
    while ((getline line < "sorted_items.txt") > 0)
        print "Read back:", line
}
$ awk -f readback_close.awk items.csv
Read back: Milk
Read back: Bread
Read back: Cheese
Read back: Rice
Read back: Paneer
Read back: Rusk
Read back: Atta
# Without close(), the file may not be visible to getline yet
$ rm -f sorted_items.txt
$ cat readback_noclose.awk
BEGIN { FS = "," }
{ print $1 > "sorted_items.txt" }
END {
    while ((getline line < "sorted_items.txt") > 0)
        print "Read back:", line
    print "Lines read: none visible, file not fully flushed"
}
$ awk -f readback_noclose.awk items.csv
Lines read: none visible, file not fully flushed
# The file exists after awk finishes, even though getline saw nothing
$ cat sorted_items.txt
Milk
Bread
Cheese
Rice
Paneer
Rusk
Atta
$ rm -f sorted_items.txt

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 output_session.py

The session split items into three files by category, then showed that print | "sort" sorted all items together. The key point is the close-before-read rule: if you write to a file in the main loop and read it back in END, call close("file") first. Without it, the output buffer may not be flushed, and getline sees nothing, as the second run proved.

Key idea

Call close("file") or close("cmd") before you read a file you wrote or before you reopen the same command with new input; otherwise the write is buffered and the read sees stale or missing data.

Checkpoint

Check: Output to files and commands

Question 1 of 2 In awk, what does print $1 > "out.txt" do on the second line that matches?
Choose one answer.

Fails because the file is already open Writes to stdout and ignores the redirect Appends to out.txt (the file was truncated on the first match) Truncates out.txt again and writes only this line

Show the answer: Question 1 of 2

Answer: Appends to out.txt (the file was truncated on the first match)

The output session showed three Dairy items written to Dairy.txt by three separate prints, all in one awk run. The file was truncated once at the first print.
Question 2 of 2 You write to a file in the main loop and read it back with getline in END, but getline returns nothing. What is the likely fix?
Choose one answer.

Add fflush() after every print Call close("file") before the getline Use print >> instead of print > Open the file with a mode flag

Show the answer: Question 2 of 2

Answer: Call close("file") before the getline

The output session showed that without close, getline in END read nothing, even though the file existed after awk finished. close flushes the buffer and resets the read position.

Multi-line programs with functions

A one-liner works for a quick filter, but a 20-line report belongs in a file. awk -f report.awk data.csv reads the program from report.awk. Comments start with # and run to the end of the line. User-defined functions use the syntax function name(params) { body }.

A program file with a function Python · program_session.py

"""awk programs: awk -f, comments, functions with local variables, and getline.

The session shows:
1. awk -f report.awk for maintainable multi-line programs
2. Comments with # (to the end of the line)
3. User-defined functions with local variables via extra parameters
4. getline acceptable use and the "cmd" | getline trap
"""
from session import run, write

write("data.csv", """\
Milk,Dairy,45
Bread,Bakery,30
Cheese,Dairy,120
Paneer,Dairy,80
Rusk,Bakery,25
""")

write("report.awk", """\
# report.awk: generate a sales report by category
# Usage: awk -f report.awk sales.csv

# User-defined function with a local variable
# The extra parameter 'i' is a local (gawk and POSIX)
function print_line(cat, items, total,   i) {
    i = total / items
    printf "%-10s %5d %8d %7.1f\\n", cat, items, total, i
}

BEGIN { FS = "," }

# For each input line, accumulate counts and sums
{ count[$2]++; sum[$2] += $3 }

END {
    # Header
    printf "%-10s %5s %8s %7s\\n", "Category", "Items", "Total", "Average"
    printf "%-10s %5s %8s %7s\\n", "--------", "-----", "-----", "-------"

    # Sort keys (gawk extension)
    PROCINFO["sorted_in"] = "@ind_str_asc"

    for (k in sum) {
        print_line(k, count[k], sum[k])
    }
}
""")

write("lines.txt", """\
header
data1
data2
""")

run("""
cat report.awk
cat data.csv

# Run the program file with awk -f
awk -f report.awk data.csv

# getline: read the next line into $0 (or into a variable)
cat lines.txt
awk 'NR==1 { getline; print "Skipped header, first data:", $0 }' lines.txt

# Read a line into a named variable
awk 'NR==1 { if ((getline nextline) > 0) print "Line 1:", $0, "| Line 2:", nextline }' lines.txt

# "cmd" | getline runs a command and reads its output
awk 'BEGIN { "date +%Y" | getline year; print "Year:", year }'

# The trap: "cmd" | getline complicates error handling
# If the command produces no output, getline returns 0; if it fails, -1
awk 'BEGIN { cmd="cat /nonexistent 2>/dev/null"; if ((cmd | getline line) <= 0) print "No output or failed"; else print "Got:", line; close(cmd) }'

# Preferred: pass data via -v or as files, not via "cmd" | getline
year=$(date +%Y)
awk -v year="$year" 'BEGIN { print "Year from shell:", year }'

rm lines.txt
""")

Output

$ cat report.awk
# report.awk: generate a sales report by category
# Usage: awk -f report.awk sales.csv

# User-defined function with a local variable
# The extra parameter 'i' is a local (gawk and POSIX)
function print_line(cat, items, total,   i) {
    i = total / items
    printf "%-10s %5d %8d %7.1f\n", cat, items, total, i
}

BEGIN { FS = "," }

# For each input line, accumulate counts and sums
{ count[$2]++; sum[$2] += $3 }

END {
    # Header
    printf "%-10s %5s %8s %7s\n", "Category", "Items", "Total", "Average"
    printf "%-10s %5s %8s %7s\n", "--------", "-----", "-----", "-------"

    # Sort keys (gawk extension)
    PROCINFO["sorted_in"] = "@ind_str_asc"

    for (k in sum) {
        print_line(k, count[k], sum[k])
    }
}
$ cat data.csv
Milk,Dairy,45
Bread,Bakery,30
Cheese,Dairy,120
Paneer,Dairy,80
Rusk,Bakery,25
# Run the program file with awk -f
$ awk -f report.awk data.csv
Category   Items    Total Average
--------   -----    ----- -------
Bakery         2       55    27.5
Dairy          3      245    81.7
# getline: read the next line into $0 (or into a variable)
$ cat lines.txt
header
data1
data2
$ awk 'NR==1 { getline; print "Skipped header, first data:", $0 }' lines.txt
Skipped header, first data: data1
# Read a line into a named variable
$ awk 'NR==1 { if ((getline nextline) > 0) print "Line 1:", $0, "| Line 2:", nextline }' lines.txt
Line 1: header | Line 2: data1
# "cmd" | getline runs a command and reads its output
$ awk 'BEGIN { "date +%Y" | getline year; print "Year:", year }'
Year: 2026
# The trap: "cmd" | getline complicates error handling
# If the command produces no output, getline returns 0; if it fails, -1
$ awk 'BEGIN { cmd="cat /nonexistent 2>/dev/null"; if ((cmd | getline line) <= 0) print "No output or failed"; else print "Got:", line; close(cmd) }'
No output or failed
# Preferred: pass data via -v or as files, not via "cmd" | getline
$ year=$(date +%Y)
$ awk -v year="$year" 'BEGIN { print "Year from shell:", year }'
Year from shell: 2026
$ rm lines.txt

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 program_session.py

The report.awk file defines print_line(cat, items, total, i). The extra parameter i after the required three is a local variable: awk fills only the arguments the caller passes, so i starts as zero and lives only inside the function. This convention is portable to every POSIX awk and avoids polluting the global namespace.

getline reads the next line into $0 (or into a named variable). The session’s NR==1 { getline; print ... } skipped a header and printed the first data line. The form "cmd" | getline var runs a command and reads its output, but it complicates error handling: a missing command or an empty result returns 0 or −1, and the program continues unless you check. The preferred alternative is passing data via -v var="$shell_value" or as a file, which keeps the control flow simple.

getline error handling

Wrong: "cmd" | getline var; print var assumes the command succeeded. Right: if (("cmd" | getline var) > 0) print var; else print "failed" > "/dev/stderr"; close("cmd"). Or avoid the issue by passing data with -v from the shell.

Checkpoint

Check: Multi-line programs with functions

Question 1 of 2 How do you declare a local variable inside an awk function?
Choose one answer.

Declare it with var inside the function body Use the local keyword before the variable name Prefix the variable name with an underscore Add it as an extra parameter after the real ones, separated by spaces

Show the answer: Question 1 of 2

Answer: Add it as an extra parameter after the real ones, separated by spaces

The program session showed function print_line(cat, items, total, i), where i is a local. The spaces before i are a convention to separate real params from locals.
Question 2 of 2 A script runs "ls /missing" | getline line and prints line. The directory does not exist. What happens?
Choose one answer.

awk prints an error and exits The script waits forever for input getline returns 0 or -1, line is empty, and awk continues line contains the error message from ls

Show the answer: Question 2 of 2

Answer: getline returns 0 or -1, line is empty, and awk continues

The program session showed that a missing file made getline return 0 (no output) and the script printed "No output or failed". The trap is that awk continues silently.

Practise: build a report from a log

Try these on your own terminal, using the data files from the sessions. Each task is one awk program (possibly in a file), and the recorded sessions show every piece you need.

From sales.csv, print the grand total of all sales in a single number. Hint: grand += $3 in the main loop, print grand in END.
From kirana.csv, print only the category with the lowest total sales. Hint: start with min_val = 99999999.
From customers.csv and orders.csv, print orders whose customer id starts with “C00”. Hint: add a pattern /^C00/ on the second file.
Split items.csv into files named <category>_items.txt (e.g. Dairy_items.txt). Hint: print $1 > ($2 "_items.txt").
Write an awk program that reads a CSV, counts rows per category, and prints them sorted by count descending. Hint: print count[k], k then pipe to sort -rn.

Interview questions

Warm-up (fresher to 2–5 years): why does for (k in array) come out in a different order on different awk versions, and how do you fix it?

The POSIX standard says the order of keys in for (k in array) is unspecified, so each awk implementation chooses its own traversal. gawk and mawk both use hash tables internally, but their hashing differs, so the same keys appear in different sequences. A script that depends on a particular order works on one machine and breaks on another.

The portable fix is to pipe the output to sort: awk '...' | sort gives a deterministic order on every awk. gawk offers PROCINFO["sorted_in"] = "@ind_str_asc" (or "@ind_num_asc" for numeric keys) to sort inside the loop itself, which avoids the extra process but is not portable to mawk or the macOS awk.

Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.

Choose a pass

Mid to senior

A billing script joins orders to customers with awk 'FNR==NR { c[$1]=$2; next } { print $0, c[$2] }' customers orders. One day a customer complains that their invoice shows no name. What went wrong, and how would you make the script report the problem instead of hiding it?

Model answer, Pro. Model answers open with any pass; prices are in the notice on this page. First step and follow-ups
Mid to senior

An awk script writes intermediate results to a temp file in the main loop, then reads them back with getline in END to produce a summary. The summary comes out empty on a small input and only partly filled on a large one. What is the likely cause, and what one-line fix makes it reliable?

Model answer, Pro. First step and follow-ups

Flashcards · 6 cards

Recall it, then check

Say the answer to yourself before you look: recalling it is what makes it stick.

How does awk group lines by key in one pass?

Its arrays are associative: count[$2]++ counts lines per value of field 2, sum[$2] += $3 adds up field 3 per key, and an END block prints the totals.
How do you make an awk join print only the lines whose key was found?

Test membership with (k in a): FNR==NR { a[$1]; next } ($2 in a) { print } prints a line of the second file only when its key was loaded, while a bare lookup gives an empty value for a missing key.
In which order does for (k in array) visit the keys?

POSIX leaves it unspecified, and gawk and mawk differ, so pipe the output to sort or use gawk's PROCINFO["sorted_in"] for a fixed order.
How does the FNR==NR idiom join two files in awk?

While FNR equals NR, awk is still reading the first file, so FNR==NR { name[$1] = $2; next } loads it into an array; the second file then looks each key up. Missing keys give empty strings, and a repeated key overwrites.
What do print > "file", print >> "file" and close() do in awk?

> empties the file once per run and then keeps writing to it, >> always appends, and print | "cmd" feeds a command. close() flushes and closes it, which must come before reading a file the program wrote.
How are longer awk programs and their functions written?

In a file run with awk -f prog.awk; a function's local variables are declared as extra parameters after its real ones.

Key takeaways

awk arrays are associative: count[$2]++ groups by field 2, sum[$2] += $3 accumulates, and (k in a) tests membership.
The for (k in array) order is unspecified: pipe to sort or use gawk’s PROCINFO["sorted_in"] for deterministic output.
The FNR==NR idiom joins two files: the first is loaded into an array, the second is looked up, but missing keys return empty and duplicate keys overwrite.
print > "file" truncates once per run; print >> "file" always appends; print | "cmd" pipes to a command; call close() before you read a file you wrote.
Multi-line programs go in a file with awk -f; local variables in functions are declared as extra parameters; prefer -v over "cmd" | getline for simpler error handling.

Terms in this lesson

awk array key: The string that identifies an entry in an awk associative array, written inside brackets: count["Dairy"] uses the key Dairy to store or look up a value.
join key: A field that appears in both files of a join, used to match rows: the FNR==NR idiom loads one file's key into an array and looks up the other file's key against it.
output field separator: The string awk prints between fields when you use print with commas; set with -v OFS=, or by assigning OFS in the program. The default is a single space.

References

GNU awk manual: Arrays in awk (Free Software Foundation)
GNU awk manual: Printing Output (Free Software Foundation)
GNU awk manual: User-defined Functions (Free Software Foundation)
POSIX.1-2024: awk utility (The Open Group and IEEE)

Report a problem with this lesson

In this module: Text processing: the Unix toolbox

cut, tr, head, tail, wc, paste and friends Free lesson, Pro answers: 5 minutes
sort and uniq: counting and top-N reports Free lesson, Pro answers: 10 minutes
grep: searching text and files Free lesson, Pro answers: 10 minutes
Regular expressions for grep, sed and awk Free lesson, Pro answers: 10 minutes
sed: substitute, delete and print Free lesson, Pro answers: 10 minutes
sed beyond s///: hold space and multiline Free lesson, Pro answers: 25 minutes
awk: fields, patterns and actions Free lesson, Pro answers: 15 minutes
awk programs: arrays, joins and reports Free lesson, Pro answers: 10 minutes
find and xargs: working with many files Free lesson, Pro answers: 15 minutes
jq: query and reshape JSON from the shell Free lesson, Pro answers: 15 minutes
CSV, TSV and YAML from the shell Free lesson, Pro answers: 10 minutes
Dates, times and non-integer numbers Free lesson, Pro answers: 20 minutes

What you will learn

Before you start

What you will learn

Before you start

Group by key with associative arrays

A formatted report with printf

Join two files with FNR and NR

Output to files and commands

Multi-line programs with functions

Practise: build a report from a log

Interview questions

A billing script joins orders to customers with awk 'FNR==NR { c[$1]=$2; next } { print $0, c[$2] }' customers orders. One day a customer complains that their invoice shows no name. What went wrong, and how would you make the script report the problem instead of hiding it?

An awk script writes intermediate results to a temp file in the main loop, then reads them back with getline in END to produce a summary. The summary comes out empty on a small input and only partly filled on a large one. What is the likely cause, and what one-line fix makes it reliable?

Key takeaways

References