Skip to content
Shell & TerminalDeep Dive Published Updated 5 min readViews unavailable

sort in Shell Pipelines: Locale, Keys, Stability, and Reproducible Output

Make sort results reproducible by defining locale, key boundaries, numeric semantics, stability, and temporary-storage behavior.

sort orders text records according to a comparison policy. The default policy is not necessarily byte order: locale collation, blank handling, numeric interpretation, key extraction, and tie-breaking can all change results. A pipeline that appears deterministic on one workstation can produce a different order in a container, CI runner, or production host with another locale. If sorted output is a cache key, manifest, comparison input, or deployment artifact, make those rules explicit.

Sorting also has an input model. Conventional sort treats newline as a record delimiter. It is not a general parser for quoted CSV, binary records, or arbitrary pathnames. GNU sort has NUL-delimited modes, while strict portability requires checking which options the target standard and implementation provide. Choose the record protocol before choosing sort flags.

Locale is part of the comparator

The LC_COLLATE category controls text ordering for many implementations. In a human locale, accented letters and case can collate differently from raw byte order. For machine-generated identifiers and stable build output, a common policy is:

LC_ALL=C sort -- input.txt

This makes comparison follow the C locale on systems with a POSIX locale implementation. It is not a universal answer for human names, linguistic ordering, or Unicode normalization. Two visually identical strings can have different code-point sequences, and locale collation does not necessarily normalize them. State whether the task is byte reproducibility or human-language ordering.

Locale can also affect character classes and numeric separators. GNU sort’s numeric mode uses locale-aware blanks and decimal punctuation. If an input field is an ASCII decimal value, pin the locale and define whether signs, empty fields, thousands separators, and exponents are allowed. Do not sort version strings, semantic versions, IP addresses, or arbitrary decimal identifiers with ordinary lexicographic order and call it numeric order.

Define keys and tie-breaking

Key selection is a data contract. With whitespace-delimited input, fields may include leading blanks and locale-sensitive characters. With an explicit separator, an empty field can be significant. Test rows with missing fields, repeated keys, empty values, and separator characters inside data. If the format permits quoting or embedded separators, parse it first rather than expecting sort to understand that grammar.

GNU sort uses a last-resort whole-line comparison when selected keys compare equal. That means rows with equal keys may be ordered by remaining text even if the key specification looks stable. The stable option suppresses that fallback comparison and preserves the original relative order of equal keys. Use it when earlier input order is meaningful, and ensure upstream order itself is deterministic.

For example, sort records by a tab-separated service key:

LC_ALL=C sort --stable -t "$(printf '\t')" -k1,1 -- services.tsv

The separator construction is shell-specific enough to deserve a test on the target shell. GNU sort supports more concise NUL and separator forms as extensions; avoid using them in portable scripts without a feature check.

Sort unique values only when equality matches policy

sort’s unique mode emits one representative for lines that compare equal under selected keys and ordering options. That equality may not mean byte-for-byte identical records. Case folding, numeric keys, dictionary order, and key ranges can cause distinct lines to compare equal. If the requirement is exact duplicate removal, use a full-record key under a pinned locale, or define canonicalization before sorting.

uniq is different: it removes adjacent repeated lines, so input normally needs a compatible sort first. Sorting can reorder records and changes which representative is kept. If first-seen wins is the requirement, preserve sequence numbers or use a data structure that records first occurrence rather than sorting away order.

Temporary storage, atomicity, and failure handling

Large sorts may use temporary files. Temporary directory, available space, filesystem quotas, and permissions affect success. Set a controlled TMPDIR or implementation-specific temporary-directory option, and monitor scratch storage. A sort can fail after reading input because temporary storage fills up; do not treat an empty output file as a valid result.

GNU sort can write output to a named file and supports some in-place forms by reading input before opening output. That is not a crash-safe transaction. The manual warns that a serious error or crash during in-place output may lose data, and merge mode can open output earlier. Safer deployment code writes to a fresh same-filesystem temporary file, checks sorting and validation status, then renames it under the application’s durability policy.

For pipelines, sort may succeed on a truncated producer stream unless pipeline failure is propagated. In Bash, pipefail helps report a failed component, but sort may already have written partial output. Stage the result and publish only if the entire pipeline completed successfully. Preserve diagnostics on stderr and do not suppress errors to make jobs appear green.

Test deterministic output

Build fixtures containing upper and lower case, accented text, composed and decomposed Unicode, empty keys, numeric-looking values, repeated keys, records without final newline, and unusual bytes permitted by the chosen format. Run under multiple locales and confirm either that output is intentionally locale-sensitive or that it remains fixed with the configured locale. Test scratch storage exhaustion and unwritable output directories in a controlled environment.

Record sort implementation and version when GNU-only options affect results. A reproducible pipeline states record delimiter, locale, key ranges, numeric model, tie behavior, duplicate definition, and publication procedure. With those conditions explicit, sort can be a reliable external-memory primitive rather than an accidental source of drift.

Related:

Sources:

Comments