GNU sort: External Merge Runs, Buffer Sizing, and Scratch-Space Planning

GNU sort can process input larger than available memory by dividing records into sorted runs and merging them. For large jobs, the key capacity questions are how many runs are created, how many can be merged at once, and whether memory, file descriptors, scratch space, and I/O can sustain the chosen settings.

The memory-behavior summary below follows the GNU Coreutils 9.11 sort manual. Coreutils 9.12 (released 2026-09-14) notes improved use of available memory and parallel operation for unknown-sized input such as pipes; do not apply the 9.11 description unchanged to pipe-fed jobs on 9.12. See the 9.12 release notes. The options below are GNU extensions; do not assume they are available with every POSIX sort implementation. Check the version and manual on the target host before using them in portable scripts.

From input records to merge passes

The normal GNU sort path reads records into a memory buffer, sorts a run, and writes that run to temporary storage when more input cannot fit. It then merges sorted runs. If there are more runs than the merge batch can handle at once, sort can merge groups into intermediate files before the final merge. Thus the job may need temporary storage for multiple generations of runs, not only one copy of the final output.

The manual exposes several controls, but each addresses a different resource:

The buffer-size option, -S or –buffer-size, sets the initial main-memory buffer size. It accepts byte counts and suffixes such as M, G, and percent. The buffer can grow when a single record is larger than the requested size, so this is not a hard memory ceiling.
The batch-size option, –batch-size=N, limits how many inputs a merge handles at a time. A larger fan-in can reduce merge generations, but it also consumes file descriptors and merge buffers. GNU sort may reduce the effective value when the process cannot open the requested number of files.
The parallel option, –parallel=N, runs multiple sorts concurrently. It can improve CPU throughput, but memory requirements rise with parallelism; the manual says using N threads increases memory usage by a factor of log N, rather than providing a free speedup.
The temporary-directory option, -T DIR, selects where temporary files are created. Repeating it supplies multiple directories. If it is not specified, GNU sort uses the applicable environment/default temporary location.

The system therefore needs a resource budget across RAM, file descriptors, temporary capacity, and I/O bandwidth. A successful choice on an SSD-backed workstation may perform badly on a network filesystem or a nearly full container volume.

Plan scratch capacity around merge work

When the run count exceeds the selected merge fan-in, GNU sort can write intermediate merge generations before producing the final stream. Peak scratch use therefore depends on input size and record distribution, the effective --batch-size, parallel jobs, and filesystem behavior; it is not reliably inferred from -S or from the final output size alone. Measure a representative workload and leave headroom for concurrent jobs and filesystem limits.

-T DIR selects where temporary files are created, and repeating it supplies additional directories. Use dedicated writable scratch locations with known capacity and I/O characteristics. A no-output capacity run against an existing test input can exercise the selected settings without replacing an output file:

sort -S 2G --parallel=4 --batch-size=8 \
  -T "$SORT_SCRATCH" -- records.tsv > /dev/null

Here SORT_SCRATCH is a pre-created writable directory reserved for the test. The values are illustrative, not recommendations; use the same ordering options as the real workload when comparing runs.

Measure resource use instead of treating flags as promises

The example settings are illustrative, not recommendations. With parallel execution, aggregate memory can exceed one nominal buffer, and oversized records can cause growth beyond the requested initial buffer. Measure peak resident memory, temporary bytes, file-descriptor use, elapsed time, and scratch-device throughput using representative input. Include skewed and unusually long records in test data.

A smaller merge batch may reduce per-pass descriptor pressure but add merge generations and I/O. A larger batch may reduce those generations while increasing simultaneous open files and buffers. Additional temporary directories help only if they have suitable capacity and independent I/O paths; listing several paths does not guarantee balanced throughput.

For already sorted inputs that use the same comparator, GNU sort’s merge mode can merge them without sorting each input again. That mode is an assertion about the inputs, not a repair option: if any input was sorted under another locale or key definition, the merged result may not satisfy the desired order.

For a useful capacity record, note the Coreutils version, input size and record distribution, buffer size, parallelism, merge batch size, descriptor limit, scratch filesystem, and measured peak use. External merging lets GNU sort handle input larger than memory; it does not guarantee that every workload fits a fixed RAM or disk budget.

Related:

Sources:

From input records to merge passes

Plan scratch capacity around merge work

Measure resource use instead of treating flags as promises

Comments