awk: fields, patterns and actions

Five lines of awk read a 10,000-row CSV and print the five-city average age in one pass: no header skipped by hand, no field counts kept in another variable, no second file written and read back. The one program awk -F, 'NR > 1 { total += $2; people += 1 } END { print people, "people, average age", total/people }' people.csv is one pattern-action pairA sentence of an awk program: a condition that decides which records it runs on, and an action in braces that runs when it does., run on every row of the file, and the result is the number the lesson will print back: 5 people, average age 37.

You will read a record’s fields with $0, $1 to $NF; count them with NR, NF and FNR; split the input with -F and FS, rebuild it with OFS and $1 = $1; pick lines with BEGIN, END, /regex/, an expression, a range and !; format a report with printf and the built-ins length, substr, split, sub, gsub and match; and predict the one surprise a learner meets on the first day, where "10" < "9" is true as text and false as numbers. Read the regular-expressions lesson first, because awk patterns use the same syntax; sed is awk’s cousin for simple substitutions.

Records, fields and the pattern-action loop

An awk program is a list of pattern { action } pairs that awk runs on every record of its input. By default a recordawk record: An input record awk reads: by default one line of input, placed in $0, with NR counting records across every file and FNR the record number within the current file. is one line, so the program { print $1 } on a two-line file runs twice, prints field 1 the first time and field 1 the second time. A fieldawk field: A piece of an awk record cut out by the field separator. $1 is the first field, $NF the last, and NF the count. Fields are 1-based; $0 is the whole record. is a piece of the record that the field separatorfield separator: awk's FS variable (and the -F option): the pattern that cuts each record into fields. A single space is the default, which means runs of white-space, with leading and trailing white-space skipped. cuts out: $1 is the first field, $NF the last, and NF is their count. $0 is the whole record, exactly as awk read it; NR counts records across every file awk reads, and FNR the record number within the current file.

The scene uses the records_session.py scenario: a CSV with a header and five data rows. The default field separator is runs of white-space, so a comma-separated line looks like one field; -F, tells awk to split on a comma, and from then on $1 is the name, $2 the age and $3 the city. NR > 1 skips the header; the final END block prints the totals the main loop built. Follow the records through the loop and watch BEGIN run before the input, END after it, and every record pass through the pattern test and the action.

"""Records, fields and the pattern-action loop on a small people.csv.

The session shows:
1. awk runs one program on every record of its input; a record is a line by default.
2. $0 is the whole record; $1, $2 to $NF are its fields; NF is the field count.
3. NR counts records across every file; FNR resets at the start of each file.
4. awk's default field separator is runs of white-space; -F sets another one (-F, for a CSV).
5. A pattern { action } pair only runs the action on records that match the pattern.
"""
from session import files, run

files("people.csv")

run("""
# A small CSV with a header: name, age and city for five people
cat people.csv

# Default field separator: white-space, so a comma-separated line looks like one field
awk '{ print NR, NF, $0 }' people.csv

# -F, tells awk to split on a comma, so now there are three fields on each row
awk -F, '{ print NR, NF, $1 }' people.csv

# $0 is the whole record; $1 is the first field and $NF is the last one
awk -F, '{ print NR, $1, "lives in", $NF }' people.csv

# Skip the header with the pattern NR > 1; the action then runs on data rows only
awk -F, 'NR > 1 { print NR, $1, "is", $2, "years old" }' people.csv

# A record for which no pattern matches is skipped; a plain action with no pattern runs on every record
awk -F, 'NR > 1 { total += $2; people += 1 } END { print people, "people, average age", total/people }' people.csv

# FNR resets at the start of each file; NR does not. Give awk the same file twice to see the difference
awk -F, 'NR > 1 { print FILENAME, NR, FNR, $1 }' people.csv people.csv
""")

# A small CSV with a header: name, age and city for five people
$ cat people.csv
name,age,city
Asha,34,Pune
Rohan,41,Kochi
Meera,28,Chennai
Vikram,37,Pune
Priya,45,Kochi
# Default field separator: white-space, so a comma-separated line looks like one field
$ awk '{ print NR, NF, $0 }' people.csv
1 1 name,age,city
2 1 Asha,34,Pune
3 1 Rohan,41,Kochi
4 1 Meera,28,Chennai
5 1 Vikram,37,Pune
6 1 Priya,45,Kochi
# -F, tells awk to split on a comma, so now there are three fields on each row
$ awk -F, '{ print NR, NF, $1 }' people.csv
1 3 name
2 3 Asha
3 3 Rohan
4 3 Meera
5 3 Vikram
6 3 Priya
# $0 is the whole record; $1 is the first field and $NF is the last one
$ awk -F, '{ print NR, $1, "lives in", $NF }' people.csv
1 name lives in city
2 Asha lives in Pune
3 Rohan lives in Kochi
4 Meera lives in Chennai
5 Vikram lives in Pune
6 Priya lives in Kochi
# Skip the header with the pattern NR > 1; the action then runs on data rows only
$ awk -F, 'NR > 1 { print NR, $1, "is", $2, "years old" }' people.csv
2 Asha is 34 years old
3 Rohan is 41 years old
4 Meera is 28 years old
5 Vikram is 37 years old
6 Priya is 45 years old
# A record for which no pattern matches is skipped; a plain action with no pattern runs on every record
$ awk -F, 'NR > 1 { total += $2; people += 1 } END { print people, "people, average age", total/people }' people.csv
5 people, average age 37
# FNR resets at the start of each file; NR does not. Give awk the same file twice to see the difference
$ awk -F, 'NR > 1 { print FILENAME, NR, FNR, $1 }' people.csv people.csv
people.csv 2 2 Asha
people.csv 3 3 Rohan
people.csv 4 4 Meera
people.csv 5 5 Vikram
people.csv 6 6 Priya
people.csv 7 1 name
people.csv 8 2 Asha
people.csv 9 3 Rohan
people.csv 10 4 Meera
people.csv 11 5 Vikram
people.csv 12 6 Priya

The session’s awk -F, 'NR > 1 { print NR, $1, "is", $2, "years old" }' people.csv printed five lines, one per data row. The totals one-liner printed 5 people, average age 37, where the integer 37 came out without a decimal because the quotient 185 / 5 is exact; a non-integer quotient would carry the default OFMT instead. Giving the same file twice to awk showed how NR keeps counting while FNR restarts at 1, which is why FNR == NR is the usual way to tell “I am reading the first file” when two files are joined.

Two traps bite on the first day. One is a header counted as a record; the other is a wrong field index, which awk never warns about because an uninitialised field is “” (empty) and 0 as a number.

Setting the separators, and rebuilding a record

The field separator decides how awk cuts a record into fields. Three ways set it: -F sep on the command line (the shortcut), -v FS=sep on the command line (the general form), and BEGIN { FS = sep } inside the program. All three take effect before the first record is read, so field 1 of record 1 is already set correctly. A plain { FS = sep } in the main action is a trap, because record 1 is split with the old FS before the assignment runs.

OFS controls what print puts between the fields it writes; it is one space by default, and setting it changes the separator between every pair of fields a comma-separated print $1, $2 writes. But awk prints $0 as it was read, so a plain print with no fields writes the input commas back, OFS or no OFS. Assigning any field (even to itself: $1 = $1) tells awk to rewrite $0 from the current fields with OFS between them, and the next print writes the rebuilt record. ORS is the output record separator, a newline by default; setting it changes what print puts after each record.

"""FS, OFS, -v and why $1=$1 rebuilds the record.

The session shows:
1. -F is a shortcut for -v FS=; both set the field separator before awk reads the input.
2. BEGIN { FS=... } also sets it in time for the first record; a plain { FS=... } in the main
   action is a trap, because record 1 is split before the assignment runs.
3. OFS controls the separator print puts between fields; it is a single space by default.
4. A plain print writes $0 as it was read, so OFS does not reach the output; an assignment such
   as $1=$1 rebuilds $0 with the current OFS between the fields.
5. ORS is the output record separator; it is a newline by default.
"""
from session import files, run

files("people.csv")

run("""
# -F, splits on commas; the action prints $1 and $3 with a single space between them (the default OFS)
awk -F, 'NR > 1 { print $1, $3 }' people.csv

# -v FS=, does the same thing: -F is a shortcut for setting FS before the input is read
awk -v FS=, 'NR > 1 { print $1, $3 }' people.csv

# -v OFS=' | ' changes the output separator; the fields print with ' | ' between them
awk -F, -v OFS=' | ' 'NR > 1 { print $1, $3 }' people.csv

# BUT: a plain print, which writes $0, shows the input commas, not the OFS
awk -F, -v OFS=' | ' 'NR > 1 { print }' people.csv

# $1=$1 forces awk to rebuild $0 from the fields with the current OFS between them
awk -F, -v OFS=' | ' 'NR > 1 { $1=$1; print }' people.csv

# BEGIN { FS=... } also works, because BEGIN runs before awk reads any record
awk 'BEGIN { FS="," } { print NR, NF, $1 }' people.csv

# A plain { FS=... } in the main action is a trap: FS changes AFTER record 1 was already split
# with the default separator, so record 1 looks like one field and only from record 2 on it has three
awk '{ FS="," } { print NR, NF, $1 }' people.csv

# -v FS=, or -F, avoids both forms of the trap: FS is set once, before the first record is read
awk -v FS=, '{ print NR, NF, $1 }' people.csv

# ORS is the output record separator; set it to a comma to join the names into one line
awk -F, -v ORS=', ' 'NR > 1 { print $1 } END { printf "\\n" }' people.csv
""")

# -F, splits on commas; the action prints $1 and $3 with a single space between them (the default OFS)
$ awk -F, 'NR > 1 { print $1, $3 }' people.csv
Asha Pune
Rohan Kochi
Meera Chennai
Vikram Pune
Priya Kochi
# -v FS=, does the same thing: -F is a shortcut for setting FS before the input is read
$ awk -v FS=, 'NR > 1 { print $1, $3 }' people.csv
Asha Pune
Rohan Kochi
Meera Chennai
Vikram Pune
Priya Kochi
# -v OFS=' | ' changes the output separator; the fields print with ' | ' between them
$ awk -F, -v OFS=' | ' 'NR > 1 { print $1, $3 }' people.csv
Asha | Pune
Rohan | Kochi
Meera | Chennai
Vikram | Pune
Priya | Kochi
# BUT: a plain print, which writes $0, shows the input commas, not the OFS
$ awk -F, -v OFS=' | ' 'NR > 1 { print }' people.csv
Asha,34,Pune
Rohan,41,Kochi
Meera,28,Chennai
Vikram,37,Pune
Priya,45,Kochi
# $1=$1 forces awk to rebuild $0 from the fields with the current OFS between them
$ awk -F, -v OFS=' | ' 'NR > 1 { $1=$1; print }' people.csv
Asha | 34 | Pune
Rohan | 41 | Kochi
Meera | 28 | Chennai
Vikram | 37 | Pune
Priya | 45 | Kochi
# BEGIN { FS=... } also works, because BEGIN runs before awk reads any record
$ awk 'BEGIN { FS="," } { print NR, NF, $1 }' people.csv
1 3 name
2 3 Asha
3 3 Rohan
4 3 Meera
5 3 Vikram
6 3 Priya
# A plain { FS=... } in the main action is a trap: FS changes AFTER record 1 was already split
# with the default separator, so record 1 looks like one field and only from record 2 on it has three
$ awk '{ FS="," } { print NR, NF, $1 }' people.csv
1 1 name,age,city
2 3 Asha
3 3 Rohan
4 3 Meera
5 3 Vikram
6 3 Priya
# -v FS=, or -F, avoids both forms of the trap: FS is set once, before the first record is read
$ awk -v FS=, '{ print NR, NF, $1 }' people.csv
1 3 name
2 3 Asha
3 3 Rohan
4 3 Meera
5 3 Vikram
6 3 Priya
# ORS is the output record separator; set it to a comma to join the names into one line
$ awk -F, -v ORS=', ' 'NR > 1 { print $1 } END { printf "\n" }' people.csv
Asha, Rohan, Meera, Vikram, Priya,

The session’s awk -F, -v OFS=' | ' 'NR > 1 { print }' people.csv wrote the five rows with commas, because print wrote $0 as it was; the next line, which added $1 = $1 before the print, wrote the five rows with | between the fields: Asha | 34 | Pune through Priya | 45 | Kochi. The { FS = "," } trap at the end of the session showed record 1 with NF = 1 (one field, the whole header line) and record 2 onwards with NF = 3.

Patterns: BEGIN, END, regex, expressions and ranges

An awk patternawk pattern: The condition before an awk action: BEGIN, END, a /regex/, an expression, or a range of two patterns. The action runs on records that match it; a plain action runs on every record. picks the records an action runs on. awk has five kinds, and they can be combined with &&, || and !:

• A BEGIN or END patternBEGIN and END pattern: Two special awk patterns that match no record: the action of BEGIN runs once before any input is read, that of END once after the last record. Common uses are setting defaults and printing totals. matches no record; its action runs once, before any input or after the last record. BEGIN is where defaults go (header line, FS and OFS); END is where totals go.
• A /regex/ pattern matches $0 against an extended regular expression. /Pune/ runs the action on every record that holds the four letters Pune anywhere.
• An expression as a pattern runs the action when the expression is non-zero or non-empty: NR > 1, $2 > 35, $3 == "Pune".
• A range pattern, written start, end, where each half is a pattern itself, matches from the first record that matches start to the first later record that matches end, inclusive on both ends.
• ! before a pattern negates it: !/Pune/ matches every record that is not matched by /Pune/.

"""Patterns: BEGIN, END, /regex/, expression, range and ! negation.

The session shows:
1. BEGIN runs once before any input; END once after the last record.
2. /regex/ as a pattern matches $0 against the regex and runs the action on records that match.
3. An expression as a pattern runs the action when the expression is non-zero and non-empty.
4. A range pattern /start/,/end/ runs from the first match of start to the first later match of end,
   inclusive on both ends.
5. ! before a pattern negates it; two !patterns and a comma do NOT make a negated range.
"""
from session import files, run

files("people.csv")

run("""
# BEGIN runs once before the input; END once after it. Here they write a header and a footer.
awk -F, 'BEGIN { print "name | age" } NR > 1 { print $1, "|", $2 } END { print "---" }' people.csv

# A /regex/ pattern matches $0 against the regex: here, every row that mentions Pune
awk -F, '/Pune/ { print }' people.csv

# An expression pattern: non-zero or non-empty runs the action. Here, people older than 35.
awk -F, 'NR > 1 && $2 > 35 { print $1, "is", $2 }' people.csv

# Two expressions can combine with && and ||, like in C. Pune AND older than 35:
awk -F, 'NR > 1 && $2 > 35 && $3 == "Pune" { print $1, "lives in Pune and is", $2 }' people.csv

# A range pattern: from the first record matching Rohan to the first later record matching Priya
awk -F, '/Rohan/,/Priya/ { print NR, $1 }' people.csv

# ! negates one pattern: print every record that does NOT match Pune
awk -F, 'NR > 1 && !/Pune/ { print }' people.csv

# BEGIN with totals: a running sum of ages, with a header, a per-row line and a total at the end
awk -F, 'BEGIN { print "roll call" } NR > 1 { total += $2; print "  " $1 } END { print "sum of ages:", total }' people.csv
""")

# BEGIN runs once before the input; END once after it. Here they write a header and a footer.
$ awk -F, 'BEGIN { print "name | age" } NR > 1 { print $1, "|", $2 } END { print "---" }' people.csv
name | age
Asha | 34
Rohan | 41
Meera | 28
Vikram | 37
Priya | 45
---
# A /regex/ pattern matches $0 against the regex: here, every row that mentions Pune
$ awk -F, '/Pune/ { print }' people.csv
Asha,34,Pune
Vikram,37,Pune
# An expression pattern: non-zero or non-empty runs the action. Here, people older than 35.
$ awk -F, 'NR > 1 && $2 > 35 { print $1, "is", $2 }' people.csv
Rohan is 41
Vikram is 37
Priya is 45
# Two expressions can combine with && and ||, like in C. Pune AND older than 35:
$ awk -F, 'NR > 1 && $2 > 35 && $3 == "Pune" { print $1, "lives in Pune and is", $2 }' people.csv
Vikram lives in Pune and is 37
# A range pattern: from the first record matching Rohan to the first later record matching Priya
$ awk -F, '/Rohan/,/Priya/ { print NR, $1 }' people.csv
3 Rohan
4 Meera
5 Vikram
6 Priya
# ! negates one pattern: print every record that does NOT match Pune
$ awk -F, 'NR > 1 && !/Pune/ { print }' people.csv
Rohan,41,Kochi
Meera,28,Chennai
Priya,45,Kochi
# BEGIN with totals: a running sum of ages, with a header, a per-row line and a total at the end
$ awk -F, 'BEGIN { print "roll call" } NR > 1 { total += $2; print "  " $1 } END { print "sum of ages:", total }' people.csv
roll call
  Asha
  Rohan
  Meera
  Vikram
  Priya
sum of ages: 185

The session’s awk -F, '/Rohan/,/Priya/ { print NR, $1 }' people.csv printed four records (NR 3 through 6: Rohan, Meera, Vikram, Priya), because the range runs from the first match of its start to the first match of its end and takes both ends. The expression pattern NR > 1 && $2 > 35 picked the three people above 35: Rohan, Vikram, Priya; combining it with && $3 == "Pune" narrowed the result to Vikram, who is the one Pune resident over 35. And !/Pune/ on the NR > 1 rows printed the three non-Pune rows: Rohan, Meera, Priya.

Built-in string functions

A short set of functions covers almost every day-to-day text job. Their shape is the same as C’s where that matches, with awk’s own 1-based positions where it does not.

• printf(fmt, ...) formats one line like C’s: %s for a string, %d for an integer, %10.2f for a 10-wide floating-point with two decimals, %-10s for a left-padded string. printf adds no newline on its own.
• length(s) is the number of characters in s; without an argument it works on $0.
• substr(s, start) and substr(s, start, len) take a slice of s starting at start, counting from 1.
• index(s, t) returns the 1-based position of t in s, or 0 when t is not there.
• split(s, a, sep) cuts s on sep, fills a[1] to a[n] and returns n; sep can be a string or a regex.
• sub(/re/, repl, s) changes the first match in s and returns 1 or 0; gsub changes every match and returns the count. The default target is $0, and awk rewrites the record when it changes.
• match(s, /re/) sets RSTART to the 1-based position of the first match (or 0 for no match) and RLENGTH to its length (or -1). Pair it with substr and a loop to walk every match in a line.
• tolower(s) and toupper(s) map the ASCII letters.

"""Built-in string functions: printf, length, substr, index, split, sub, gsub, match, tolower, toupper.

The session shows:
1. printf formats output as C's does: %s for strings, %d for integers, %-12s for a 12-wide left pad.
2. length(s) and substr(s, start, len): length is characters, substr starts at 1 (not 0).
3. index(s, t): the position of t in s, 1-based, or 0 when t is not there.
4. split(s, a, sep): breaks s on sep, fills a[1]..a[n] and returns n.
5. sub(/re/, repl, s) changes the first match in s and returns 1 or 0; gsub changes every match
   and returns the count. The default target is $0.
6. match(s, /re/) sets RSTART (1-based position, 0 for no match) and RLENGTH (-1 for no match).
7. tolower(s) and toupper(s) map the ASCII letters.
"""
from session import files, run

files("people.csv")

run("""
# printf with column widths: %-12s left-pads a string, %3d a number
awk -F, 'NR > 1 { printf "%-10s %3d  %s\\n", $1, $2, $3 }' people.csv

# length is the number of characters; substr is a 1-based slice
awk 'BEGIN { s="Chennai"; print length(s), substr(s, 1, 3), substr(s, 4) }'

# index returns the first position of a substring, or 0 when it is not there
awk 'BEGIN { print index("Kochi", "ch"), index("Kochi", "xy") }'

# split returns the number of fields it produced, and fills the given array
awk 'BEGIN { n = split("Asha,34,Pune", a, ","); for (i = 1; i <= n; i++) print i, a[i] }'

# sub changes the first match on the record and returns 1 when it matched, 0 otherwise
awk -F, 'NR > 1 { if (sub(/Pune/, "Pune (MH)")) print "changed:", $0; else print "same:   ", $0 }' people.csv

# gsub changes every match; its return value is the count. Default target is $0 (and the record is rebuilt)
awk -F, 'NR > 1 { OFS="|" } NR > 1 { c = gsub(/i/, "I"); $1 = $1; print c, $0 }' people.csv

# match sets RSTART and RLENGTH; a loop can walk every pin from a line of comma-separated pairs
awk 'BEGIN { s = "city=Pune, pin=411001, city=Kochi, pin=682001"; while (match(s, /pin=[0-9]+/)) { print substr(s, RSTART, RLENGTH); s = substr(s, RSTART + RLENGTH) } }'

# tolower and toupper on the ASCII letters
awk -F, 'NR > 1 { printf "%-10s %s\\n", toupper($1), tolower($3) }' people.csv
""")

# printf with column widths: %-12s left-pads a string, %3d a number
$ awk -F, 'NR > 1 { printf "%-10s %3d  %s\n", $1, $2, $3 }' people.csv
Asha        34  Pune
Rohan       41  Kochi
Meera       28  Chennai
Vikram      37  Pune
Priya       45  Kochi
# length is the number of characters; substr is a 1-based slice
$ awk 'BEGIN { s="Chennai"; print length(s), substr(s, 1, 3), substr(s, 4) }'
7 Che nnai
# index returns the first position of a substring, or 0 when it is not there
$ awk 'BEGIN { print index("Kochi", "ch"), index("Kochi", "xy") }'
3 0
# split returns the number of fields it produced, and fills the given array
$ awk 'BEGIN { n = split("Asha,34,Pune", a, ","); for (i = 1; i <= n; i++) print i, a[i] }'
1 Asha
2 34
3 Pune
# sub changes the first match on the record and returns 1 when it matched, 0 otherwise
$ awk -F, 'NR > 1 { if (sub(/Pune/, "Pune (MH)")) print "changed:", $0; else print "same:   ", $0 }' people.csv
changed: Asha,34,Pune (MH)
same:    Rohan,41,Kochi
same:    Meera,28,Chennai
changed: Vikram,37,Pune (MH)
same:    Priya,45,Kochi
# gsub changes every match; its return value is the count. Default target is $0 (and the record is rebuilt)
$ awk -F, 'NR > 1 { OFS="|" } NR > 1 { c = gsub(/i/, "I"); $1 = $1; print c, $0 }' people.csv
0|Asha|34|Pune
1|Rohan|41|KochI
1|Meera|28|ChennaI
1|VIkram|37|Pune
2|PrIya|45|KochI
# match sets RSTART and RLENGTH; a loop can walk every pin from a line of comma-separated pairs
$ awk 'BEGIN { s = "city=Pune, pin=411001, city=Kochi, pin=682001"; while (match(s, /pin=[0-9]+/)) { print substr(s, RSTART, RLENGTH); s = substr(s, RSTART + RLENGTH) } }'
pin=411001
pin=682001
# tolower and toupper on the ASCII letters
$ awk -F, 'NR > 1 { printf "%-10s %s\n", toupper($1), tolower($3) }' people.csv
ASHA       pune
ROHAN      kochi
MEERA      chennai
VIKRAM     pune
PRIYA      kochi

The session’s awk 'BEGIN { s="Chennai"; print length(s), substr(s, 1, 3), substr(s, 4) }' printed 7 Che nnai, which is the length, the first three characters, and the slice from position 4 to the end. The match loop walked "city=Pune, pin=411001, city=Kochi, pin=682001" and printed pin=411001 and pin=682001 by trimming the string after each find. printf "%-10s %3d %s\n", $1, $2, $3 on the data rows produced a clean five-row table with the names left-padded to ten characters and the ages right-padded to three.

Numbers, strings and comparisons

awk reads every field as text, but a field that looks like a number compares as one when its sibling in the comparison is also a numeric string or a number constant. The three cases pull in different directions, and the one that stings is the comparison between a numeric field and a string constant: the string constant wins, and the comparison is text, so $1 < "50" on a column of ages sorts 10 before 9.

"""Numbers, strings and comparisons: where the quiet surprises live.

The session shows:
1. awk reads every field as text, but a field that looks like a number compares as a number
   when its sibling in the comparison is also a numeric string or a constant number.
2. STRING CONSTANTS stay strings: "10" < "9" is true, because the byte 1 comes before 9.
3. Forcing numeric with +0 makes the comparison use numbers.
4. An uninitialised variable is "" as a string and 0 as a number.
5. awk numbers are double-precision floats: 1/3 prints 0.333333 by default; OFMT controls the format.
6. A whole-number value prints without a decimal point by default (OFMT applies to non-integers).
"""
from session import run

run("""
# A sort by field 1 as text puts "10" before "9", because '1' is a smaller byte than '9'
printf '10\\n9\\n2\\n100\\n' | sort

# awk comparing two STRING CONSTANTS does the same: it is a byte-wise compare
awk 'BEGIN { if ("10" < "9") print "yes, 10 < 9 as strings"; else print "no" }'

# Forcing both sides to be numbers with +0 flips the answer: 10 < 9 is false
awk 'BEGIN { if ("10"+0 < "9"+0) print "10 < 9 numeric"; else print "9 < 10 numeric" }'

# A field from the input is a NUMERIC STRING when it looks like a number; the compare is numeric
printf '10\\n9\\n2\\n100\\n' | awk '{ if ($1 < 50) print $1, "under 50" }'

# But comparing a field to a string CONSTANT forces a string compare, so "10" < "50" as text
printf '10\\n9\\n2\\n100\\n' | awk '{ if ($1 < "50") print $1, "under 50" }'

# Uninitialised variables are "" as strings and 0 as numbers
awk 'BEGIN { print "x+0 =", x+0; print "<" x ">"; print "x==\\"\\":", (x == "") }'

# awk numbers are IEEE 754 double-precision floats
awk 'BEGIN { print 1/3; print 2/3; print 1/7 }'

# A whole-number value prints without a decimal point (OFMT applies to non-integers only)
awk 'BEGIN { print 2+2; print 4/2; print 10/4 }'

# OFMT formats fractional results; the default is "%.6g"
awk 'BEGIN { OFMT = "%.3f"; print 1/3 }'

# CONVFMT (not OFMT) is used when a number is turned into a string by concatenation
awk 'BEGIN { CONVFMT = "%.2f"; s = "" (1/3); print "s is <" s ">" }'
""")

# A sort by field 1 as text puts "10" before "9", because '1' is a smaller byte than '9'
$ printf '10\n9\n2\n100\n' | sort
10
100
2
9
# awk comparing two STRING CONSTANTS does the same: it is a byte-wise compare
$ awk 'BEGIN { if ("10" < "9") print "yes, 10 < 9 as strings"; else print "no" }'
yes, 10 < 9 as strings
# Forcing both sides to be numbers with +0 flips the answer: 10 < 9 is false
$ awk 'BEGIN { if ("10"+0 < "9"+0) print "10 < 9 numeric"; else print "9 < 10 numeric" }'
9 < 10 numeric
# A field from the input is a NUMERIC STRING when it looks like a number; the compare is numeric
$ printf '10\n9\n2\n100\n' | awk '{ if ($1 < 50) print $1, "under 50" }'
10 under 50
9 under 50
2 under 50
# But comparing a field to a string CONSTANT forces a string compare, so "10" < "50" as text
$ printf '10\n9\n2\n100\n' | awk '{ if ($1 < "50") print $1, "under 50" }'
10 under 50
2 under 50
100 under 50
# Uninitialised variables are "" as strings and 0 as numbers
$ awk 'BEGIN { print "x+0 =", x+0; print "<" x ">"; print "x==\"\":", (x == "") }'
x+0 = 0
<>
x=="": 1
# awk numbers are IEEE 754 double-precision floats
$ awk 'BEGIN { print 1/3; print 2/3; print 1/7 }'
0.333333
0.666667
0.142857
# A whole-number value prints without a decimal point (OFMT applies to non-integers only)
$ awk 'BEGIN { print 2+2; print 4/2; print 10/4 }'
4
2
2.5
# OFMT formats fractional results; the default is "%.6g"
$ awk 'BEGIN { OFMT = "%.3f"; print 1/3 }'
0.333
# CONVFMT (not OFMT) is used when a number is turned into a string by concatenation
$ awk 'BEGIN { CONVFMT = "%.2f"; s = "" (1/3); print "s is <" s ">" }'
s is <0.33>

The session’s awk 'BEGIN { if ("10" < "9") print "yes, 10 < 9 as strings"; else print "no" }' printed yes, 10 < 9 as strings, because the first byte of "10" is 1 and the first byte of "9" is 9, and 1 comes before 9 in the byte table. The next line, with +0 on each side, printed 9 < 10 numeric. On a column of four values, { if ($1 < 50) print $1, "under 50" } kept 10, 9 and 2 and dropped 100, as the numeric compare asked. The same filter with "50" as a string constant kept 10, 2 and 100 — because "100" < "50" is true as text — and dropped 9.

Three more facts the session shows. An uninitialised variable is "" as a string and 0 as a number, so total += $2 without ever setting total is safe. awk numbers are IEEE 754 double-precision floats, so print 1/3 writes 0.333333 by default; OFMT and CONVFMT change the format (OFMT for print on a non-integer, CONVFMT for a number turned into a string by concatenation). A whole-number value, such as 185 / 5 or 2 + 2, prints without a decimal point: OFMT applies to the fractional ones only.

"""gawk 5.4.1 (on this track) and macOS's /usr/bin/awk: the same POSIX core, with extensions.

The session shows:
1. --version works on both; the names and the maintainers differ, so a one-liner tells you which awk ran.
2. The POSIX core (patterns, actions, FS, OFS, NR, NF, printf, sub, gsub, length, substr, split, match)
   runs the same on both; this session's lines come from the same scripts on each awk.
3. gawk has extension functions that macOS's awk does not. gensub, a non-destructive gsub, is one
   of them: gawk returns the new string; macOS awk reports it as an undefined function.
4. --posix on gawk turns gawk-only extensions off, so you can test what POSIX awk alone would do.
"""
from session import run

run("""
# awk --version on this track's awk (GNU Awk, from the GNU gawk project)
awk --version | head -1

# The same option on macOS's own awk (One True AWK, from the Brian Kernighan tree)
/usr/bin/awk --version

# The POSIX core runs the same on both: NR, NF, $1, length
printf 'alpha beta\\ngamma delta epsilon\\n' | awk '{ print NR, NF, length($0), $1 }'
printf 'alpha beta\\ngamma delta epsilon\\n' | /usr/bin/awk '{ print NR, NF, length($0), $1 }'

# gensub is a gawk extension: it returns the result and does not change the original string
awk 'BEGIN { s = "x and x"; print gensub(/x/, "y", 1, s); print "s =", s }'

# macOS's awk does not have gensub, and it says so
!/usr/bin/awk 'BEGIN { print gensub(/x/, "y", 1, "x and x") }'

# --posix on gawk turns the gawk-only extensions off (gensub, strftime, PROCINFO ...)
!awk --posix 'BEGIN { print gensub(/x/, "y", 1, "x and x") }'
""")

# awk --version on this track's awk (GNU Awk, from the GNU gawk project)
$ awk --version | head -1
GNU Awk 5.4.1, API 4.1, PMA Avon 8-g1
# The same option on macOS's own awk (One True AWK, from the Brian Kernighan tree)
$ /usr/bin/awk --version
awk version 20200816
# The POSIX core runs the same on both: NR, NF, $1, length
$ printf 'alpha beta\ngamma delta epsilon\n' | awk '{ print NR, NF, length($0), $1 }'
1 2 10 alpha
2 3 19 gamma
$ printf 'alpha beta\ngamma delta epsilon\n' | /usr/bin/awk '{ print NR, NF, length($0), $1 }'
1 2 10 alpha
2 3 19 gamma
# gensub is a gawk extension: it returns the result and does not change the original string
$ awk 'BEGIN { s = "x and x"; print gensub(/x/, "y", 1, s); print "s =", s }'
y and x
s = x and x
# macOS's awk does not have gensub, and it says so
$ /usr/bin/awk 'BEGIN { print gensub(/x/, "y", 1, "x and x") }'
/usr/bin/awk: calling undefined function gensub
 source line number 1
[exit status 2]
# --posix on gawk turns the gawk-only extensions off (gensub, strftime, PROCINFO ...)
$ awk --posix 'BEGIN { print gensub(/x/, "y", 1, "x and x") }'
awk: cmd. line:1: warning: regexp constant for parameter #1 yields boolean value
awk: cmd. line:1: fatal: function `gensub' not defined
[exit status 2]

Two awks run on this computer. awk (on this track) is GNU Awk 5.4.1, the gawk project; /usr/bin/awk on macOS is the One True AWK (awk version 20200816 in the session), the Brian Kernighan tree that macOS ships with its own name. The POSIX core runs the same on both, as the session’s awk '{ print NR, NF, length($0), $1 }' showed on a two-line input. The two differ on extensions: gensub, a non-destructive gsub that returns the new string, works on gawk and fails on /usr/bin/awk with calling undefined function gensub. Running gawk with --posix turns the gawk extensions off, which is the test for “would POSIX awk alone do this?”.

A field with a comma inside quotes, as a CSV writes an address that contains a comma, is the one case awk -F, cannot handle on its own: on Asha,"Pune, Maharashtra", awk counts three fields, not two, and splits the city across $2 and $3. The next lesson, CSV, TSV and YAML from the shell, handles the quoted-comma case with a CSV-aware reader; use that or gawk 5’s --csv option when the input is proper CSV.

Practise: write the awk you reach for

Try these in your terminal, from the examples/ folder. Each is one awk command, and the recorded sessions show every piece you need.

1. On people.csv, print one line per row that reads <name>, from <city>, with the header skipped. Hint: NR > 1 and printf.
2. Print the number of people in each city, as a short table. Hint: a count variable and print in END; the next lesson, awk arrays and reports, will do it with one array in one line.
3. Rewrite people.csv as a TSV: tab-separated, one row per line. Hint: -F,, -v OFS='\t' and the field rebuild.
4. Print every row whose name starts with a capital P or a capital V. Hint: a regex pattern on $1, with ~ for “matches”.
5. Print the average age of the Pune residents only. Hint: an expression pattern on $3, a running total, a running count and a division in END.

Work out each one before you open its page. The questions below use records awk did not show in the lesson, but every answer follows from a rule above:

Interview questions

Warm-up: what are NR, NF and $NF in an awk program (fresher to 2–5 years)?

Three of awk’s built-in names, and the three you reach for first. NR is the record number across all the input awk is reading, counting from 1; it is 6 after awk has read every line of the six-line people.csv and 12 after awk has read it twice (giving the file to awk twice on the command line). NF is the number of fields in the current record, after awk has split it on FS; it is 3 for every row of a comma-separated people.csv with -F,, and 1 if the same file is read with the default white-space separator, because none of its lines holds a blank or a tab. $NF is the last field of the current record, the field whose index is the value of NF; on Asha,34,Pune with -F, it is Pune. Two cousins are useful too: FNR is the record number within the current file and resets at the start of each file, so FNR == NR is true only while awk is still reading the first file; and $0 is the whole record as awk read it. All five hold their values until the next record is read; a BEGIN block sees them with their starting values (NR and NF are 0), and an END block sees them with the values they had after the last record.

The two deeper questions below are about a column total that printed the wrong number, and a sort that put 10 before 9. Work out your own answer before you open each one:

Key takeaways

• An awk program is a list of pattern { action } pairs; for each record awk reads, it runs the actions whose pattern matched, with BEGIN before the input and END after it.
• $0 is the whole record, $1 to $NF are the fields, NF is their count, and NR and FNR count the records; -F sep, -v FS=sep and BEGIN { FS=sep } all set the field separator in time.
• OFS sets what print $1, $2 puts between fields; a plain print writes $0 as it was read, and an assignment such as $1 = $1 is how you rebuild $0 with the current OFS between the fields.
• A pattern can be BEGIN, END, a /regex/, an expression, a range start, end of two patterns or any of those negated with !; patterns combine with && and || as in C.
• awk reads every field as a numeric string when it looks like a number, but a string constant on the other side of a comparison forces a byte-wise compare — so "10" < "9" is true as text, false as numbers, and +0 settles the ambiguity when the input is already numbers.