Skip to content
Shell & TerminalDeep Dive Published Updated 6 min readViews unavailable

GNU split: Chunk Boundaries, Record Preservation, and Exact Reconstruction

Partition large files with GNU split while preserving the intended byte or record contract, validating reconstruction, and isolating failed output.

Splitting a file sounds like a storage operation, but the important decision is semantic: must a chunk preserve bytes, complete records, or an assignment of records to workers? GNU split implements several different contracts. Selecting the wrong mode can create syntactically invalid chunks, lose the original ordering during reconstruction, or consume unexpected temporary disk space.

This article targets GNU Coreutils. The split shipped with macOS and the versions in BSD systems have different option sets. On a Mac with GNU Coreutils installed through a package manager, the command may be named gsplit; inspect its version and help rather than assuming that /usr/bin/split accepts GNU options. The examples use disposable output directories and an immutable input file. A file being appended to or replaced during partitioning requires a separate snapshot or publication protocol.

Choose the partition unit before the size

split -b SIZE partitions a byte stream. It can divide a UTF-8 character, a text line, a compressed stream, or an application record. This is appropriate when every piece will be reassembled before interpretation. It is inappropriate when each piece must be independently parsed as JSON, a media file, or a database export.

split -l COUNT counts newline-delimited lines by default. A final line without a terminating newline remains part of the input; line partitioning does not repair or normalize it. This mode suits plain one-record-per-line data when a newline cannot occur inside a record. A CSV file with quoted embedded newlines does not satisfy that premise. Parse such a file with a CSV-aware tool before selecting record partitions.

split -C SIZE prefers complete lines while keeping each output within the size bound. A single line longer than the bound is divided between files. That last rule is crucial: -C is not an unconditional record-preservation mode. A downstream parser requiring whole records must reject oversized records, accept larger chunks, or use an application-specific splitter.

For fixed-count output, -n N partitions using input size and may split lines. -n l/N keeps lines intact while targeting approximately equal byte regions, so output sizes can vary greatly. One unusually long record can cover a region and leave another chunk empty. Equal file count does not imply equal processing cost.

Keep generated output separate from unrelated files

A predictable prefix is useful for naming, but a shared directory introduces stale-output hazards. If yesterday’s run produced more chunks than today’s run, globbing all matching names can append old data during reconstruction. Use a fresh directory for every run, and publish that directory only after validation.

The following Bash example partitions an existing immutable artifact.bin into 64 MiB chunks. Numeric suffixes are fixed at six digits; the command fails if that namespace is exhausted. The prefix contains no user-controlled shell expression.

#!/usr/bin/env bash
set -euo pipefail
export LC_ALL=C

input=artifact.bin
work=$(mktemp -d)
printf 'Output directory: %s\n' "$work"

split --bytes=64M --numeric-suffixes=0 --suffix-length=6 \
  --additional-suffix=.part -- "$input" "$work/chunk-"

cat "$work"/chunk-*.part > "$work/reassembled.bin"
cmp -- "$input" "$work/reassembled.bin"

GNU’s M multiplier is binary, while MB is decimal. State the intended unit in an operational runbook rather than leaving readers to infer it from a human-readable label. The reconstruction order is the filename order in the C locale, and fixed-width numeric suffixes preserve numerical order in that ordering.

This example retains output for inspection. In an automated pipeline, add cleanup policy that distinguishes success from failure and never removes the original input. If a split fails halfway through, partial chunks remain; do not label the directory complete. A separate success marker or an atomic directory publication step can communicate completion to consumers.

Partition records for independent workers

For an immutable newline-delimited event export, a line-count partition is easy to reason about:

split --lines=50000 --numeric-suffixes=0 --suffix-length=6 \
  --additional-suffix=.ndjson -- events.ndjson "$work/events-"

This requires that each line actually be a complete JSON record. split does not validate JSON or ensure consistent schemas. Have each worker report its input chunk name, records accepted, records rejected, and output artifact. The controller should reject a missing chunk even if every worker that did run reported success.

An alternative, split -n r/4, distributes complete lines round-robin among four outputs. This can distribute similarly sized records without knowing total input size first. However, concatenating the outputs does not restore the original input order. If ordering matters, carry a stable sequence number and reconstruct by that number with an explicit sorting contract, or choose contiguous line partitions instead.

Do not use round-robin assignment to assume balanced CPU time. A tiny record may trigger expensive processing while a larger one may be trivial. Measure per-chunk duration and distinguish storage partitioning from scheduling policy.

Account for temporary storage and transformed output

The non-round-robin fixed-count modes need input size. When input is a pipe or another source whose size cannot readily be determined, GNU split first copies the input into temporary storage. Set and monitor TMPDIR if the temporary filesystem is small. A streaming command that worked with -l can therefore fail with -n despite identical final output volume.

--filter sends each chunk through a shell command instead of writing it directly. GNU sets the FILE environment variable to the corresponding output filename. This enables per-chunk compression, but it adds shell quoting, filter exit status, and compressed-format validation to the pipeline. Avoid interpolating untrusted text into the filter command. Remember that compressed output has a different byte contract: concatenate decompressed chunk content in order and compare it against the original, rather than expecting a simple comparison of compressed files to establish correctness.

Keep stderr separate from data output. A verbose progress message belongs in logs, not in a manifest consumed as filenames. Treat filenames as data and use an explicit manifest if the workflow later allows spaces or unusual characters.

Verify more than a plausible chunk count

For byte-preserving contiguous partitions, cmp against reconstructed output is a direct acceptance check. A nonzero result must be interpreted: the files may differ, or comparison itself may have failed. A cryptographic digest is useful for transport integrity, but a locally generated digest does not prove that the input was authentic. Preserve a trusted upstream signature or manifest when authenticity matters.

For independently processed records, compare aggregate counts and business-level invariants as well as file hashes. Check empty input, a final unterminated line, a record larger than the byte target, too few records for the requested chunks, suffix exhaustion, a full filesystem, and a failed worker. Test round-robin output separately because it intentionally changes the concatenation order.

If validation fails, quarantine the run directory, inspect the first mismatch or failing worker, and rerun from a stable input. Never repair a failed partition by silently dropping its final chunk or mixing pieces from different runs. Chunking is reliable when the partition rule, ordering rule, completion signal, and reconstruction check all describe the same input artifact.

Related:

Sources:

Comments