GNU csplit: Split Files at Verified Record Boundaries

GNU csplit partitions an input stream at line-number or regular-expression boundaries. That makes it useful for records whose beginnings are marked in the data itself: transaction logs, mail-style archives, sectioned reports, or files with explicit headers. It is not the same tool as GNU split: split chooses byte, line, or record counts, while csplit reacts to the content of input lines.

This article describes GNU Coreutils csplit, not every BSD implementation. The examples were checked with GNU Coreutils 9.12. On macOS, Homebrew installs the command as gcsplit because the formula prefixes commands that macOS already provides; verify with gcsplit --version and gcsplit --help before using GNU-specific flags. GNU/Linux packages commonly expose GNU csplit as csplit, though the installed version depends on the distribution.

Treat patterns as a boundary program

The invocation accepts an input file (or - for standard input) followed by one or more patterns. A positive integer pattern splits before that line number. /REGEXP/[OFFSET] writes the current section up to but not including the next line matching the regular expression; an optional signed offset adjusts the boundary. %REGEXP%[OFFSET] uses a similar search to skip a section without creating an output file. {N} repeats the preceding pattern N additional times; {*} repeats it as many times as possible until input is exhausted.

By default the program writes files named xx00, xx01, and so on, and prints the byte count of each file. -f selects a prefix, -n selects the number of suffix digits, -b supplies a printf-style suffix format, and -s suppresses the byte-count messages. Do not interpret those counts as record counts: they are bytes written to each output file.

Here is a small example with records beginning at a literal marker:

gcsplit --quiet --prefix=part- --digits=4 --suppress-matched \
  events.log '/^BEGIN TRANSACTION /' '{*}'

{*} asks the prior regular-expression pattern to repeat until no more input can be split. --suppress-matched removes the matching marker lines from the later output pieces; that is useful only when those boundary lines are metadata you intentionally do not want in the result. Remove that option if each record needs to retain its header. Because the input begins with a BEGIN line in this example, the first attempted section is empty; add --elide-empty-files (-z) if that empty file is not useful.

csplit does not infer your record grammar. The regular expression must match the line form your producer emits. A marker with an optional timestamp, a leading prefix, or changed capitalization can fail to match. Extra trailing bytes also matter when the expression is anchored to the end of a line; for example, a carriage return before the newline can prevent a full-line match unless the expression accounts for it. Test representative input from every producer version before using a pattern in a release or incident pipeline.

Make output isolated and failure visible

The default output prefix can collide with files from an earlier run. Use a fresh private directory and an explicit prefix so stale pieces cannot be mistaken for current output:

set -u
input=${1:?usage: split-records INPUT}
[[ -f $input ]] || { printf 'not a regular file: %s\n' "$input" >&2; exit 2; }

workdir=$(mktemp -d) || exit 1
trap 'rm -rf -- "$workdir"' EXIT

gcsplit --quiet --elide-empty-files --prefix="$workdir/record-" \
  --digits=5 --suppress-matched "$input" \
  '/^BEGIN TRANSACTION /' '{*}' || exit

printf 'pieces are staged under %s until this script exits\n' "$workdir"

The snippet deliberately uses Bash features ([[ ... ]] and EXIT), and the GNU command name shown is the Homebrew/macOS form; on a GNU/Linux host use csplit if that is the installed command. The temporary directory belongs to the current process and is removed when the script exits. If pieces must outlive the script, replace the cleanup with a deliberately managed destination, verify that it is new and private, and define retention and failure handling. Removing directory entries is not secure erasure of storage.

By default, GNU csplit removes output files it created when it encounters an error or receives common termination signals. That keeps an incomplete run from looking like a successful set of pieces, but it also removes evidence useful for debugging. -k / --keep-files changes that policy: partial files remain after an error. If you use -k, tag the directory as incomplete and publish or consume it only after the command exits successfully and all expected checks pass. Never let a later run infer completeness merely from the presence of a prefix.

The no-match behavior depends on the repeat count. With {*}, reaching end of input without another matching line is normal termination: even an input with no markers can return success and leave one output containing the unsplit input. With a finite {N} on a required /REGEXP/ split, a missing required match is an error. Therefore, test both cases and validate the output count or expected headers against the producer’s contract. A zero exit status alone does not prove that every logical record was represented or that data was not intentionally skipped by %REGEXP% or --suppress-matched.

Offsets, skipped lines, and empty pieces

Offsets are powerful but easy to misread. A positive offset moves the split boundary past the matching line by the specified number of lines; a negative offset moves it before the match. Lines inside a negative offset are not matched by subsequent regular-expression patterns. That means offset-based patterns are stateful over the remaining input rather than independent searches over the original file. Use offsets only when the format requires them, and build a tiny fixture that prints every output byte before relying on them in production.

The %REGEXP% form discards the skipped region from the output set. --suppress-matched also changes reconstruction: the boundary line is not emitted in later pieces. The files therefore cannot be concatenated to recreate the input exactly. For a lossless partition, use patterns that create only boundaries, do not request --suppress-matched, and do not skip input with %...%.

--elide-empty-files suppresses zero-byte outputs, which commonly appear when a delimiter marks the first line of a section. The output sequence remains consecutive; do not use a piece number as a source line number. Conversely, without -z, an empty file may be semantically meaningful, such as when a delimiter appears where a record body was expected. Choose explicitly rather than using -z as a generic cleanup option.

Verify both content and the failure cases

For a lossless partition, reconstruct in a fresh staging file and compare bytes:

set -euo pipefail
shopt -s nullglob
pieces=("$workdir"/record-*)
(( ${#pieces[@]} > 0 ))
cat "${pieces[@]}" > "$workdir/rejoined"
cmp --silent -- "$input" "$workdir/rejoined"

With --suppress-matched, -z, or a %...% pattern, equality is not the right test because the command deliberately changed the output. Instead assert the intended invariants: every expected header appears in exactly one file or is deliberately removed, each file begins at an allowed boundary, the count is within an expected range, and a representative payload from each source version remains intact. If record ordering matters, inspect the sort order of generated names; a fixed-width suffix makes lexical order align with the numeric sequence.

Build test input that covers a marker at the first line, at the last line, two adjacent markers, no marker, a marker-like line that should not match, and a large section. Assert that a finite required pattern fails when a marker is missing. Separately assert that {*} may succeed on a no-marker fixture and that your own record-count check rejects the resulting one-piece output when the input contract promises multiple records. Keep malformed-input fixtures alongside the script so a future regex edit cannot silently turn a valid log into one giant output file.

The source file itself must be stable during the run. If another process can append, replace, or rewrite it while it is being split, capture an immutable snapshot or use a producer protocol that closes and publishes the completed file atomically. Ensure the temporary destination has enough free space: output pieces generally contain the source bytes again, and a later reconstruction needs additional capacity. For sensitive records, restrict the staging directory, avoid printing sensitive contents in diagnostics, and apply the data-retention policy to every piece and failure artifact.

Use csplit when content-defined boundaries are the actual requirement. Use split when the boundary is a byte/line budget or a deliberate round-robin partition. In both cases, make the partition contract testable: specify whether delimiters stay with the preceding or following section, define what a missing delimiter means, isolate output, and verify a representative reconstruction before publishing the pieces.

Related:

Sources:

Treat patterns as a boundary program

Make output isolated and failure visible

Offsets, skipped lines, and empty pieces

Verify both content and the failure cases

Comments