GNU sed's Hold Space: Build Bounded Multiline Transformations Without Losing the Cycle
Use sed's pattern and hold buffers to build small multiline state machines, while testing portability, line limits, and input boundaries explicitly.
Most sed scripts transform one input line at a time, but the utility has a second buffer that can carry state across cycles. The current line is the pattern space. The hold space survives between input cycles until the script replaces or changes it. Commands such as h, H, x, g, and G move or combine those buffers; N, P, and D can deliberately make a pattern space contain multiple lines.
This is a small stream-processing model, not a magic parser. A script that accumulates an unbounded file in the hold space can use unbounded memory. A script that appends the next line and waits for a delimiter can also grow without bound when malformed input omits that delimiter. Make the state machine explicit, bound the amount of retained text, and choose a parser designed for the input format when the grammar exceeds a small line-oriented transformation.
Know what survives each cycle
At the beginning of an ordinary cycle, sed reads a line, removes its terminating newline, and places the text in pattern space. Unless a command changes control flow, the pattern space is printed by default and then discarded before the next line. The hold space is retained. Under -n, automatic printing is disabled, so every intended output must be emitted explicitly.
The basic buffer commands have distinct effects:
| Command | Effect |
|---|---|
h |
Replace hold space with a copy of pattern space |
H |
Append a newline and pattern space to hold space |
g |
Replace pattern space with a copy of hold space |
G |
Append a newline and hold space to pattern space |
x |
Exchange pattern and hold spaces |
N |
Append the next input line, separated by a newline, to pattern space |
P |
Print the first line in pattern space |
D |
Delete the first line in pattern space and restart the cycle when an embedded newline exists |
The case is meaningful: h replaces, while H appends; g replaces, while G appends. D is not the same as d: with a multiline pattern it discards only through the first embedded newline and restarts the script without reading a new input line. Confusing these operations is a common source of scripts that duplicate records, lose lines, or never make progress.
Example: emit each adjacent two-line window
This POSIX-compatible script prints every neighboring pair as a two-line block. It stores the previous line, appends the current line, prints the pair, and then resets the hold space to the current line:
sed -n '1h; 1!{H; x; p; x; h;}' input.txtFor input red, green, blue, it emits the records (red, green) and (green, blue). The initial 1h seeds the state and emits nothing. On later cycles, H appends the current line to the previous one in hold space; x makes that pair the pattern space; p emits it; the second x restores the current line to pattern space; and h makes that line the next previous value. The state remains one input line plus one current line rather than accumulating the whole file.
Before using a multiline script on production text, test a one-line file, an empty file, blank records, a final line without a newline, and an input containing delimiter-like text. sed processes text lines, not arbitrary binary records; NUL bytes and implementation-specific multibyte behavior are outside a portable text-processing contract. If blank lines are meaningful, the example preserves them but may produce visually ambiguous output, so add explicit delimiters or use a length-aware format.
Understand N, P, and D as a sliding-window machine
The GNU manual demonstrates a compact sliding-window pattern using N, P, and D: append a new line to pattern space, process or print its first line, then remove that first line and restart. For example, a two-line lookahead can inspect a neighboring line without storing the whole input. The script must prove that each loop either consumes another input line or exits. A branch that repeats without changing state is an infinite loop; a branch that executes N until a nonexistent marker appears grows the pattern space indefinitely.
When searching across a pair, the newline in pattern space is data that the regular expression must account for. In GNU sed, some extensions make it convenient to match a newline; POSIX scripts should use the commands and expression forms the standard guarantees. Do not copy a GNU-only \s, \w, -z, or address extension into a portability-critical script without documenting the dependency and testing the target implementation.
Portability and capacity are part of the design
POSIX.1-2024 specifies minimum pattern and hold space capacity of at least 8,192 bytes. That is a portability floor, not a promise that every implementation accepts arbitrarily large records. GNU sed documents no fixed built-in line-length limit beyond available memory, but a script that depends on that property is a GNU-specific design. Keep a strict bound on any record or context you retain, and fail or route large structured documents to an appropriate parser.
Set LC_ALL=C only when byte-oriented, locale-independent matching is the intended behavior. Locale can affect character classes, ranges, and multibyte interpretation. This is especially important if the input contains pathnames or localized text. A deterministic test should state its locale rather than silently changing production semantics.
Keep transformation output separate from file replacement. The hold-space model answers how a stream is transformed; it does not make sed -i portable or transactional. Validate the transformed output, check the expected record count or marker, and only then use the project’s established atomic-replacement workflow. A successful sed exit status does not prove that the script matched the intended records.
Test the script as a state machine
Build fixtures that cover zero, one, two, and several input lines; empty and repeated records; blank lines; missing final newline; very long records near the supported boundary; and malformed data that omits an expected terminator. Run them on each promised implementation, such as GNU sed on Linux and the platform’s BSD-derived sed on macOS. If the script requires a GNU extension, invoke the declared GNU binary instead of inferring it from the operating-system name.
For every branch, write down the current input line, pattern space, hold space, and control-flow destination. Confirm that a loop advances or terminates and that every line is either emitted, intentionally discarded, or retained for the next bounded window. Add a golden-output test and preserve the test input when changing a regular expression or command order. These checks make a short sed program reviewable instead of relying on visual intuition about punctuation.
Hold space is most useful for small state: a previous line, a two-line decision, or a bounded window. If the script needs nested records, escaping rules, arbitrary-length state, or schema validation, use a parser whose data model expresses those requirements directly.
Related:
- sed In-Place Editing: Portability, Backups, and Safe File Replacement
- AWK Records and Fields: Parse Text Without Pretending It Is CSV
Sources: