Bash & the POSIX Shell Module 11 – Text processing: the Unix toolbox
cut, tr, head, tail, wc, paste and friends
Slice columns and characters with cut, transform text with tr, take the first or last lines with head and tail, count lines with wc, and join files with paste.
What you will learn
- Select columns and characters from text with cut
- Transform and delete characters with tr
- Extract lines from the start or end of a file with head and tail
- Count lines, words and characters with wc
- Join files side by side with paste
Before you start
What you’ll learn · Before you start
What you will learn
- Select columns and characters from text with cut
- Transform and delete characters with tr
- Extract lines from the start or end of a file with head and tail
- Count lines, words and characters with wc
- Join files side by side with paste
Before you start
On this page
A three-gigabyte server log holds the answer to one question: which IP addresses sent the most requests yesterday? The log has 12 fields per line, and you need only two of them. One pipeline of small tools finds the answer in seconds: cut selects the fields, grep filters the date, sort orders them, uniq -c counts, and head takes the top ten.
This lesson shows you the tools that turn raw text into the columns, lines and counts you actually want. You will select fields with cut, transform characters with tr, grab the first or last lines with head and tail, count lines with wc, and join files side by side with paste. Each tool does one job; the pipeline connects them.
Selecting columns and characters with cut
cut extracts pieces of each line: columns separated by a delimiter: A character that separates fields in a line of text, such as a comma in CSV or a tab. cut uses -d to set the delimiter and defaults to a tab., or ranges of character positions. The -d option sets the delimiter, and -f picks the fields by number:
"""cut selects fields and characters from text."""
from session import write, run
write("people.csv", """\
name,email,city
alice,alice@example.com,Pune
bob,bob@example.com,Mumbai
carol,carol@example.com,Pune
dan,dan@example.com,Bengaluru
""")
run("""
# Select fields 1 and 3 from a comma-separated file
cat people.csv
cut -d, -f1,3 people.csv
# Select field 2 onwards (drop the first field)
echo "name,email,city" | cut -d, -f2-
# Select characters 1 to 10 from each line
echo "2024-03-12 09:15:23 INFO Server started" | cut -c1-19
""")Output
# Select fields 1 and 3 from a comma-separated file $ cat people.csv name,email,city alice,alice@example.com,Pune bob,bob@example.com,Mumbai carol,carol@example.com,Pune dan,dan@example.com,Bengaluru $ cut -d, -f1,3 people.csv name,city alice,Pune bob,Mumbai carol,Pune dan,Bengaluru # Select field 2 onwards (drop the first field) $ echo "name,email,city" | cut -d, -f2- email,city # Select characters 1 to 10 from each line $ echo "2024-03-12 09:15:23 INFO Server started" | cut -c1-19 2024-03-12 09:15:23
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 cut_session.py
The session shows three ways to use cut. First, -d, -f1,3 takes the first and third fields from a comma-separated line; the names and cities appeared without the emails. Second, -f2- takes field two onwards, which drops the header’s first column. Third, -c1-10 takes the first ten characters of each line, which works when every line has fixed-width columns.
Fields are numbered from 1, not 0, and -f2-4 means fields 2 through 4. A missing delimiter leaves the whole line as field 1, so cut -d: -f1 /etc/passwd prints only the user names, one per line.
The drawing shows the whole scene; its 7 steps are listed below.
Every step as text
- Raw data. The CSV file has a header row and hundreds of data rows. Each row has name, email and city fields separated by commas.
- Skip the header. tail -n +2 outputs from line 2 onwards, so the header never reaches the next stage. Only the data rows flow through.
- Extract one column. cut -d, -f3 splits each line at commas and keeps only field 3, the city. The output is now one city name per line.
- Sort the cities. sort puts identical city names next to each other. This is required for uniq, which only compares adjacent lines.
- Count each city. uniq -c counts how many times each city appears. It outputs a count followed by the city name for each unique value.
- Take the top 3. head -n 3 keeps only the first three lines. A preceding sort -rn would put the highest counts first; then head shows the top 3.
- Result. The pipeline answered the question. Three lines show the most frequent cities and their counts, extracted from a file of thousands of rows.
cut cannot handle quoted CSV fields
cut -d, -f2 breaks when a field contains a comma inside quotes, because cut sees only the delimiter character, not the quoting. For proper CSV parsing, use awk, csvtool, or a language with a CSV library.
When you know your data has no quoted fields, cut is fast and simple. Keep this rule in mind:
Key idea
Use cut -d DELIM -f N to pick columns from delimited text, and cut -c RANGE to pick character positions from fixed-width data.
Checkpoint
Check: Selecting columns and characters with cut
Transforming and deleting characters with tr
tr translates or deletes characters. It reads only from standard input, so you pipe data into it or redirect a file. The first argument is the set of characters to change, and the second is what they become:
"""tr transforms and deletes characters."""
from session import run
run("""
# Convert lowercase to uppercase
echo "hello world" | tr '[:lower:]' '[:upper:]'
# Delete all digits
echo "Order 12345 shipped" | tr -d '0-9'
# Squeeze repeated spaces into one
echo "Name City Score" | tr -s ' '
""")Output
# Convert lowercase to uppercase $ echo "hello world" | tr '[:lower:]' '[:upper:]' HELLO WORLD # Delete all digits $ echo "Order 12345 shipped" | tr -d '0-9' Order shipped # Squeeze repeated spaces into one $ echo "Name City Score" | tr -s ' ' Name City Score
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 tr_session.py
The first command turned every lowercase letter into uppercase: tr '[:lower:]' '[:upper:]' maps the POSIX character class. The second command deleted all digits with tr -d '0-9'. The third used tr -s ' ' to squeeze repeated spaces into one, which tidies columns separated by variable whitespace.
Character ranges work as you expect: 'a-z' means the letters a through z, and '0-9' means the digits. The [:class:] forms follow POSIX: [:alpha:] for letters, [:digit:] for digits, [:space:] for whitespace. The -d option deletes characters instead of translating them, and -s squeezes runs of repeated characters into one.
tr reads stdin only
tr does not take file names as arguments: use tr '[:lower:]' '[:upper:]' < file.txt, not tr '[:lower:]' '[:upper:]' file.txt.
Checkpoint
Check: Transforming and deleting characters with tr
Extracting lines from the start or end with head and tail
head takes the first lines of a file, and tail takes the last. By default both show 10 lines; -n N changes that number:
"""head and tail select lines by position."""
from session import write, run
write("data.txt", """\
header
row1
row2
row3
row4
row5
footer
""")
run("""
cat data.txt
# First 3 lines
head -n 3 data.txt
# Last 2 lines
tail -n 2 data.txt
# Starting from line 2 (skip the header)
tail -n +2 data.txt
# Skip header and drop footer: combine tail and head
tail -n +2 data.txt | head -n 5
""")Output
$ cat data.txt header row1 row2 row3 row4 row5 footer # First 3 lines $ head -n 3 data.txt header row1 row2 # Last 2 lines $ tail -n 2 data.txt row5 footer # Starting from line 2 (skip the header) $ tail -n +2 data.txt row1 row2 row3 row4 row5 footer # Skip header and drop footer: combine tail and head $ tail -n +2 data.txt | head -n 5 row1 row2 row3 row4 row5
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 head_tail_session.py
The session demonstrated three ways to slice lines. head -n 3 took the first three lines. tail -n 2 took the last two. tail -n +2 started at line 2, which skips one header line. The last command combined them: tail -n +2 | head -n 5 skips the header and takes the next five rows.
The sign on the number matters for tail: -n 5 means the last five lines, while -n +5 means starting from line five. GNU head accepts -n -5 to mean all but the last five lines, but BSD head (on macOS) does not support negative counts. For portable scripts, pipe through head or sed to drop trailing lines.
tail -f follows a growing log
tail -f /var/log/syslog keeps reading as new lines arrive. Press Ctrl+C to stop. Use tail -F to follow even when the file is rotated and replaced.
The essential forms to remember for skipping headers and taking ranges:
Key idea
head -n N and tail -n N take lines from each end; tail -n +N starts at line N, and head -n -N stops N lines before the end.
Checkpoint
Check: Extracting lines with head and tail
Counting lines, words and characters with wc
wc counts lines, words and bytes. Without options it shows all three; -l, -w and -c select one:
"""wc counts lines, words and bytes."""
from session import write, run
write("sample.txt", """\
line one text here
line two more text
line three has words
line four content now
""")
# Note: wc output format varies by platform; we show lines, words, bytes
run("""
# Count lines, words and bytes
wc sample.txt
# Count only lines
wc -l sample.txt
# Demonstrate the newline-counting gotcha
printf 'one\\ntwo' | wc -l
printf 'one\\ntwo\\n' | wc -l
""")Output
# Count lines, words and bytes
$ wc sample.txt
4 16 81 sample.txt
# Count only lines
$ wc -l sample.txt
4 sample.txt
# Demonstrate the newline-counting gotcha
$ printf 'one\ntwo' | wc -l
1
$ printf 'one\ntwo\n' | wc -l
2
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 wc_session.py
The first command counted 4 lines, 16 words and 81 bytes in the file. The second printed only the line count with -l. The third showed that wc -l counts newline characters: a string ending without a newline reports one fewer line than you might expect.
When you pipe data into wc -l, the count appears alone: grep ERROR log.txt | wc -l prints just the number, without a file name. That number is easy to capture in a variable: count=$(grep ERROR log.txt | wc -l).
wc counts newlines, not lines
printf 'one\ntwo' has no newline after the second word, so wc -l reports 1 line instead of 2. Add a trailing newline when you create text, or use grep -c '' to count non-empty lines.
Checkpoint
Check: Counting with wc
Joining files side by side with paste
paste combines lines from two or more files into columns, separated by a tab by default. The -d option sets the delimiter:
"""paste joins files side by side."""
from session import write, run
write("cities.txt", """\
Pune
Mumbai
Bengaluru
""")
write("populations.txt", """\
3100000
12400000
8400000
""")
run("""
# Join two files line by line (tab-separated by default)
paste cities.txt populations.txt
# Join with a comma delimiter
paste -d, cities.txt populations.txt
# Transpose each file into one line
paste -s cities.txt
paste -sd, cities.txt
""")Output
# Join two files line by line (tab-separated by default) $ paste cities.txt populations.txt Pune 3100000 Mumbai 12400000 Bengaluru 8400000 # Join with a comma delimiter $ paste -d, cities.txt populations.txt Pune,3100000 Mumbai,12400000 Bengaluru,8400000 # Transpose each file into one line $ paste -s cities.txt Pune Mumbai Bengaluru $ paste -sd, cities.txt Pune,Mumbai,Bengaluru
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 paste_session.py
The first command joined cities.txt and populations.txt line by line, with a tab between them. The second added -d, to produce comma-separated output. The third used -s to turn each file into one line of joined columns.
paste expects the files to have the same number of lines. If one is shorter, the missing fields appear as empty columns. For merging files on a common key rather than by line number, use join, which matches lines by a field value like a database join.
paste -sd joins a column into one line
paste -sd, turns a file of one word per line into a comma-separated list on one line: echo -e 'a\nb\nc' | paste -sd, prints a,b,c.
When you need to merge files column by column or turn rows into a single line:
Key idea
paste file1 file2 merges line 1 of each, then line 2, and so on; -d, sets the delimiter and -s transposes columns into rows.
Checkpoint
Check: Joining files with paste
Putting the tools together in a pipeline
Each tool handles one step, and a pipeline chains them. The next session builds a small report from a CSV file: skip the header, select two columns, count rows per city, and show the top three:
"""A pipeline that answers a question about the data."""
from session import write, run
write("people.csv", """\
name,email,city
alice,alice@example.com,Pune
bob,bob@example.com,Mumbai
carol,carol@example.com,Pune
dan,dan@example.com,Bengaluru
eve,eve@example.com,Mumbai
frank,frank@example.com,Mumbai
grace,grace@example.com,Pune
henry,henry@example.com,Chennai
""")
run("""
cat people.csv
# Question: which cities appear most often?
# Pipeline: skip header, extract city column, count each city, show top 3
tail -n +2 people.csv | cut -d, -f3 | sort | uniq -c | sort -rn | head -n 3
""")Output
$ cat people.csv name,email,city alice,alice@example.com,Pune bob,bob@example.com,Mumbai carol,carol@example.com,Pune dan,dan@example.com,Bengaluru eve,eve@example.com,Mumbai frank,frank@example.com,Mumbai grace,grace@example.com,Pune henry,henry@example.com,Chennai # Question: which cities appear most often? # Pipeline: skip header, extract city column, count each city, show top 3 $ tail -n +2 people.csv | cut -d, -f3 | sort | uniq -c | sort -rn | head -n 3 3 Pune 3 Mumbai 1 Chennai
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 pipeline_session.py
The pipeline answered a question in four stages. tail -n +2 skipped the header. cut -d, -f3 extracted the city column. sort | uniq -c counted each city. sort -rn | head -n 3 showed the top three by count. Each stage’s output became the next stage’s input, and the final answer printed in under a second.
Build a pipeline one stage at a time and check the output after each. When the chain grows longer than three or four stages, consider moving to awk or a small script, where the logic is easier to read and debug.
Checkpoint
Check: Putting tools together in a pipeline
Practise
Practice that adapts to you
Practise this lesson
6 questions about 6 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
Warm-up (fresher to 2–5 years): how does cut -f2 differ from cut -c2, and when would you use each?
cut -f2 selects the second delimited field, using a tab as the default delimiter. cut -c2 selects the second character of each line. Use -f when your data has columns separated by a known character, such as a CSV or /etc/passwd. Use -c when each column occupies a fixed range of characters, as in old mainframe reports or fixed-width exports. Mixing them up gives wrong results: -c2 on a comma-separated file returns just the second character, not the second field.
Work out your own first move on each of the following, then compare it with the first step on its page:
Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.
-
Fresher to mid
A script processes a CSV file by piping it straight to awk, but the header row appears in the output as if it were data. How would you exclude the header, and what are two different approaches?
-
Mid to senior
A deployment script counts active servers with curl | grep UP | wc -l. The API returns a JSON array, and the script works in staging. In production, where some hosts are down, it occasionally reports one fewer server than expected. What could cause the off-by-one, and how would you fix it?
Flashcards · 6 cards
Recall it, then check
Say the answer to yourself before you look: recalling it is what makes it stick.
-
How do cut -f2 and cut -c2 differ?
cut -f2 takes the second delimited field, with a tab as the default delimiter and -d to set another. cut -c2 takes the second character of each line, which suits fixed-width data.
-
Why does cut -d, give wrong columns on some CSV files?
cut sees only the delimiter character, not the quoting, so a comma inside a quoted field splits that field in two. Quoted CSV needs a parser that understands quotes.
-
What does tr do, and where does it read its input?
It transforms or deletes single characters, and -s squeezes repeated characters into one. It reads only standard input, so a file goes in with < file.
-
How do you print a file without its header line?
tail -n +2 file starts at line 2. head -n N and tail -n N take N lines from the start or the end.
-
Why can wc -l report one line fewer than the file seems to have?
wc -l counts newline characters, so a last line without a trailing newline is not counted.
-
How do paste and join differ?
paste joins files side by side line by line, with -d for the separator and -s to turn columns into rows. join merges two sorted files on a matching key field.
Key takeaways
cut -d DELIM -f FIELDSselects columns from delimited text, andcut -c RANGEselects fixed-width positions.trtransforms or deletes characters and reads only from standard input;-ssqueezes repeated characters into one.head -n Nandtail -n Ntake lines from each end;tail -n +Nstarts at line N, skipping a header.wc -lcounts newline characters, so a file without a trailing newline reports one fewer line than expected.pastejoins files line by line;joinmerges sorted files on a matching key field.- Build pipelines one stage at a time, checking output at each step before adding the next.
Terms in this lesson
- delimiter
- A character that separates fields in a line of text, such as a comma in CSV or a tab. cut uses -d to set the delimiter and defaults to a tab.
- text filter
- A program that reads text from standard input, transforms it, and writes the result to standard output; examples include cut, tr, head, tail, sort, uniq and grep.
References
- GNU Coreutils Manual: cut (Free Software Foundation)
- GNU Coreutils Manual: tr (Free Software Foundation)
- GNU Coreutils Manual: head (Free Software Foundation)
- GNU Coreutils Manual: tail (Free Software Foundation)
- GNU Coreutils Manual: wc (Free Software Foundation)
- GNU Coreutils Manual: paste (Free Software Foundation)
- POSIX.1-2024: cut (The Open Group and IEEE)
- man7.org: Linux man-pages (Linux man-pages project)
Report a problem with this lesson
Kept only in this browser. Your Learn progress