sed beyond s///: hold space and multiline

A configuration file wraps long lines with a trailing backslash, and every tool that reads it unwraps them first. sed can join those lines before anything else sees them, but s/// sees one line at a time and has nowhere to put a line it wants to remember. The hold spacehold space: A second buffer that sed keeps across cycles; the h, H, g, G and x commands copy or exchange data between it and the pattern space. It starts empty. is that somewhere: a second buffer that sed never touches unless you ask, where a script can stash a line and bring it back later.

You will move text between the pattern space and the hold space with h, H, g, G and x, read several input lines into one pattern space with N, carve the first line out again with P and D, jump around a script with labels and branches, and insert, append or replace whole lines. The examples use BSD sed (macOS) for portability; many scripts work the same on GNU sed 4.10, but labels and the one-line insert form need script files on BSD sed.

The hold space: h, H, g, G and x

sed reads each input line into the pattern spacepattern space: sed's working buffer. At the start of every cycle sed reads a line from the input into it (without the trailing newline), applies every command whose address matches, and prints it at the end of the cycle unless -n is given. More in “sed: substitute, delete and print”, runs the script, prints what remains (unless -n was passed), clears the pattern space and starts the next line. The hold space is empty at the start and keeps its value across lines. Five commands copy or swap text between the two:

"""Hold space commands: h, H, g, G, x"""

from session import run, write

write("lines.txt", "one\ntwo\nthree\n")

run("""
# Input file
cat lines.txt

# Reverse lines: 1!G;h;$!d
sed '1!G;h;$!d' lines.txt

# Double-space: G appends hold (empty) + newline
sed 'G' lines.txt | head -6

# Print last line only with hold space
sed -n '$p' lines.txt
""")

# Input file
$ cat lines.txt
one
two
three
# Reverse lines: 1!G;h;$!d
$ sed '1!G;h;$!d' lines.txt
three
two
one
# Double-space: G appends hold (empty) + newline
$ sed 'G' lines.txt | head -6
one

two

three

# Print last line only with hold space
$ sed -n '$p' lines.txt
three

The session showed the classic reverse-file idiom: sed '1!G;h;$!d' printed three, two, one from an input of one, two, three. On line 1, 1!G does not run, h copies one to the hold space, and $!d deletes the pattern space so nothing prints. On line 2, G appends the hold space (one) after a newline, so the pattern space becomes two\none; h copies that to the hold space; $!d deletes again. On line 3, the last, G makes the pattern space three\ntwo\none; $!d does not run, so the whole thing prints.

The same building blocks assemble in other ways. sed -n '$p' prints only the last line. sed '$!d' does the same by deleting every line that is not the last. sed -n '1!G;h;$p' reverses lines and prints only at the end. sed 'G' double-spaces a file by appending the (empty) hold space plus a newline to every line.

Reading across lines: N, P and D

Some patterns span more than one line. A trailing backslash means “the next line continues this one”; a blank line separates paragraphs; a multi-line block of configuration runs from <VirtualHost> to </VirtualHost>. The N command reads the next input line and appends it to the pattern space with an embedded newline. P prints up to the first newline. D deletes up to the first newline and restarts the cycle without reading new input if anything remains.

"""Multiline commands: N, P, D"""

from session import run, write

write("pairs.txt", "a\nb\nc\nd\n")

run("""
# Input file for joining
cat pairs.txt

# Join every two lines with +
sed 'N;s/\\n/+/' pairs.txt

# Protect last line on odd-count files
printf 'a\\nb\\nc\\n' | sed '$!N;s/\\n/+/'

# Sliding window: print with N;P;D (each pair)
printf 'a\\nb\\nc\\n' | sed 'N;P;D'

# N appends, showing embedded newline
printf 'first\\nsecond\\n' | sed 'N; l'
""")

# Input file for joining
$ cat pairs.txt
a
b
c
d
# Join every two lines with +
$ sed 'N;s/\n/+/' pairs.txt
a+b
c+d
# Protect last line on odd-count files
$ printf 'a\nb\nc\n' | sed '$!N;s/\n/+/'
a+b
c
# Sliding window: print with N;P;D (each pair)
$ printf 'a\nb\nc\n' | sed 'N;P;D'
a
b
# N appends, showing embedded newline
$ printf 'first\nsecond\n' | sed 'N; l'
first$
second$
first
second

The session showed sed 'N;s/\n/+/' joining pairs of lines with a +: a and b became a+b, and c and d became c+d. N read the second line and left the pattern space as a\nb; s/\n/+/ turned the embedded newline into a plus. When there is no next line (an odd-numbered file), N quits the script without printing, so sed '$!N;s/\n/+/' protects the last line: the three-line input a b c became a+b and c.

P and D are the inverse. sed 'N;P;D' is a two-line sliding window: every pair of adjacent lines passes through the pattern space. The session showed three lines going through the window: a and b printed a, then b and c printed b, and the final c was lost because N quit when there was no fourth line. D keeps the second line for the next cycle, where N appends the third, and so on. A continuation-line joiner works the same way: when a line ends with a backslash, N reads the next, s/\\\n// removes the backslash and the embedded newline, and a branch jumps back to try again.

Branches and labels: b, t and the loop

sed scripts run from top to bottom on each line, but two commands break the flow. b label jumps to :label unconditionally (or to the end if no label is given). t label jumps only if a substitution has succeeded since the last input line or the last t. Together they build loops and state machines.

"""Branch commands: labels, b and t"""

from session import run, write

write("dashes.txt", "a--b---c\nx--y\n")
write("data.txt", "# comment\nfoo bar\nbaz foo\n")
write("loop.sed", ":loop\ns/--/-/\nt loop\n")
write("skip.sed", "/^#/b\ns/foo/FOO/\n")

run("""
# File with multiple dashes
cat dashes.txt

# Replace all -- with - using a t loop (script file)
cat loop.sed
sed -f loop.sed dashes.txt

# File with comments
cat data.txt

# Skip substitution on comment lines with b (script file)
cat skip.sed
sed -f skip.sed data.txt

# Conditional processing: only non-blank lines
printf 'a\\n\\nb\\n' | sed -e '/^$/b' -e 's/^/> /'
""")

# File with multiple dashes
$ cat dashes.txt
a--b---c
x--y
# Replace all -- with - using a t loop (script file)
$ cat loop.sed
:loop
s/--/-/
t loop
$ sed -f loop.sed dashes.txt
a-b-c
x-y
# File with comments
$ cat data.txt
# comment
foo bar
baz foo
# Skip substitution on comment lines with b (script file)
$ cat skip.sed
/^#/b
s/foo/FOO/
$ sed -f skip.sed data.txt
# comment
FOO bar
baz FOO
# Conditional processing: only non-blank lines
$ printf 'a\n\nb\n' | sed -e '/^$/b' -e 's/^/> /'
> a

> b

The session’s label loop (stored in a script file loop.sed) replaced -- with a single - until none remained: a--b---c became a-b-c. The script :loop on one line, s/--/-/ on the next and t loop on the third set up a label, tried the substitution, and branched back to it only when the substitution succeeded. When nothing changed, the branch did not fire and sed moved on. t tests the flag set by the most recent s; two substitutions in a row clear and set the same flag.

b is the unconditional jump. The session showed a script file skip.sed with /^#/b and s/foo/FOO/ on separate lines: comment lines jumped to the end and printed unchanged, while other lines had foo replaced with FOO. b without a label means “jump to the end of the script, print and read the next line”. That is the simplest guard: the line still prints, but the rest of the script does not run on it.

Inserting, appending and changing lines: a, i and c

Sometimes a script must add text rather than transform what is already there. a text appends text after the current line is printed. i text inserts text before the current line. c text replaces the current line with text. In POSIX, the text follows a backslash and a newline (so a POSIX insert on a blank line reads: /^$/i\ on one line, --- on the next).

GNU sed accepts a one-line form, 2i inserted, but the browser runner and older versions need the backslash form. All three commands queue their text and print it at the right moment; the pattern space is unchanged by a and i, but c clears it and starts the next cycle.

"""Insert, append and change: a, i, c"""

from session import run, write

write("data.txt", "alpha\nbeta\ngamma\n")
# POSIX form requires backslash-newline for a, i, c
write("insert.sed", "1i\\\nheader\n")
write("append.sed", "$a\\\nfooter\n")
write("change.sed", "2c\\\nreplaced\n")

run("""
# Input file
cat data.txt

# Insert header before first line (POSIX form)
cat insert.sed
sed -f insert.sed data.txt

# Append footer after last line
cat append.sed
sed -f append.sed data.txt

# Change (replace) line 2
cat change.sed
sed -f change.sed data.txt
""")

# Input file
$ cat data.txt
alpha
beta
gamma
# Insert header before first line (POSIX form)
$ cat insert.sed
1i\
header
$ sed -f insert.sed data.txt
header
alpha
beta
gamma
# Append footer after last line
$ cat append.sed
$a\
footer
$ sed -f append.sed data.txt
alpha
beta
gamma
footer
# Change (replace) line 2
$ cat change.sed
2c\
replaced
$ sed -f change.sed data.txt
alpha
replaced
gamma

The session showed sed scripts in files using the POSIX backslash-and-newline form: insert.sed with 1i\ and header on the next line inserted header before the first line; append.sed with $a\ and footer appended footer after the last; and change.sed with 2c\ and replaced replaced the second line entirely. GNU sed accepts a one-line form (2i inserted), but BSD sed on macOS requires the backslash form or a script file.

Practise: join, split and transform

Work through these tasks in your terminal, using the commands from this lesson. Check your answers below.

1. Reverse the lines of a file without tac. Expected command: sed '1!G;h;$!d' file.
2. Join every two lines with a colon. Expected: sed 'N;s/\n/:/'.
3. Double-space a file (add a blank line after every line). Expected: sed G file.
4. Remove all consecutive duplicate lines (like uniq). Hint: N;/^\(.*\)\n\1$/!P;D.
5. Join lines that end with a backslash to the next line. Hint: :a; /\\$/N; s/\\\n//; ta.

Interview questions

Warm-up (fresher to 2–5 years): what are the two buffers sed uses, and when does each clear?

sed has a pattern space and a hold space. The pattern space is where sed loads each input line, runs the script, and (unless -n) prints what remains; it clears automatically before each new input line. The hold space starts empty and keeps its value until the script changes it, which is what makes it useful for remembering earlier lines. h and H copy or append to the hold space; g and G copy or append from it; x swaps the two.

Think through the following scenarios before you open the answer page:

Key takeaways

• The hold space is a second buffer that sed never touches unless the script says so; h, H, g, G and x move text between it and the pattern space.
• N appends the next line to the pattern space with an embedded newline, P prints up to the first newline and D deletes up to it and restarts the cycle.
• b label jumps unconditionally, t label jumps only when a substitution succeeded since the last line; together they build loops such as :loop; s/--/-/; t loop.
• a text, i text and c text insert, append or replace lines; the one-line form is GNU-only, so portable scripts use the backslash-and-newline form.
• When a sed script grows past a few commands, awk is often clearer.