GNU shuf: Sampling Records, Replacement Semantics, and Reproducible Runs

GNU shuf produces a random permutation of input records or selects records randomly with replacement. It is useful for test-case ordering, sampling operational exports, and distributing a work list whose original order would bias a test. The command does not decide what a record represents, whether repeated values are duplicates, or whether a sample supports a valid statistical conclusion.

This article targets GNU Coreutils. On macOS, a package-manager installation may expose the command as gshuf; the system tools should not be assumed to provide GNU behavior. Record the implementation and version when the sample becomes evidence in a benchmark, investigation, or release decision.

Define the sampling population

By default, each input line is an occurrence in the population. If an export contains the same customer ID ten times, those are ten selectable occurrences, not one unique customer. Sampling the export can therefore give that ID more weight than an ID occurring once. This can be correct for sampling events and wrong for sampling distinct accounts.

Decide whether the unit is a row, unique key, session, host, or another entity before invoking shuf. If the population needs deduplication, normalize and deduplicate using the intended identity fields first. Removing identical full lines does not establish unique entity keys, and it can discard legitimate repeated events.

Keep filters and exclusions in the sample record. A command may choose fairly among the lines it receives while the upstream query excludes the difficult cases. Random order does not repair a biased population, stale export, or incomplete time interval.

Without replacement preserves occurrences

The default mode returns a permutation: each input occurrence appears once. -n COUNT limits the result to at most that many records. If fewer input records exist, the command does not invent additional ones to fill the requested count.

The following Bash fixture demonstrates a sample of two occurrences from a three-record population. It asserts the output count and that each sampled record belongs to the population; it intentionally does not assert one random ordering.

#!/usr/bin/env bash
set -euo pipefail
export LC_ALL=C
work=$(mktemp -d)

printf '%s\n' alpha beta gamma > "$work/population"
shuf -n 2 -- "$work/population" > "$work/sample"
test "$(wc -l < "$work/sample")" -eq 2
sort "$work/sample" > "$work/sample.sorted"
sort "$work/population" > "$work/population.sorted"
test -z "$(comm -23 "$work/sample.sorted" "$work/population.sorted")"
test "$(sort -u "$work/sample" | wc -l)" -eq 2

The fixture uses simple unique ASCII labels and newline-terminated records. The set-membership check alone would not catch repeated selection if the implementation were incorrect, so it also checks that two distinct labels were selected. In a real dataset containing duplicate lines, distinct input occurrences can still produce repeated output values legitimately; validate occurrence counts against that population instead of requiring unique output values.

Use shuf -e when the population is an explicit argument list. Quote every argument whose spaces must remain within one record. Shell word splitting before the command runs changes the population. Do not build a string containing shell syntax and evaluate it merely to supply candidate records.

With replacement changes the question

-r permits selecting the same occurrence repeatedly. With -n, it produces a bounded replacement sample. Without an output count, repeat mode continues indefinitely, which is unsuitable for an unattended command that is expected to finish.

shuf --repeat --head-count=20 --echo red green blue
shuf --repeat --head-count=100 --input-range=1-6

The first command produces twenty selections from three supplied labels. The second models one hundred selections from an inclusive integer range. These are illustrative randomized experiments, not predicted outputs. They do not guarantee equal observed counts in a finite sample, and forcing equal counts afterward would change the experiment.

For load tests, explain whether replacement can select the same target concurrently. A random host list without replacement and repeated independent host selection create different traffic patterns. If a test must enforce per-host concurrency or cooldown, implement those scheduling rules separately rather than assuming that random selection enforces them.

Preserve framing for filenames and binary-safe records

Newline-delimited records work only when newline is not part of a record. For filenames, use NUL framing end to end with GNU shuf -z. Bash command substitution cannot retain NUL bytes, so do not capture that stream into a shell variable.

find ./fixtures -type f -print0 | shuf -z -n 10 > "$work/selected-files.nul"

This command requires the fixture directory to exist. Its output is a file of NUL-separated paths, not a human-readable list of ten lines. Use NUL-aware consumers and compare record counts with a compatible parser. A path containing a newline remains one filename even though a text viewer may render it across several lines.

The sample is still a snapshot of names rather than opened file handles. A file can change or disappear between enumeration and use. Work against a stable fixture tree or verify the identity/content at consumption time when the test requires that guarantee.

Reproduce the run with evidence

--random-source=FILE lets GNU shuf obtain randomness from a specified byte source. That is useful for controlled experiments, but a small source can be exhausted. Its contents also are not the same abstraction as a numeric seed accepted by some programming-language libraries.

For reproducibility, retain the input bytes, random source where policy permits, exact command/options, Coreutils version, locale, and the resulting ordered sample. Do not promise identical results across every future implementation just because the random-source file is unchanged; algorithm and random-byte consumption can change. Keeping the actual sample is the strongest replay artifact for a task scheduler.

Randomizing a work list is not a documented password-generation or token-generation contract. If the feature needs security-sensitive randomness, use an API and construction designed for that purpose. A statistical test utility selecting among public labels does not become a secret generator by replacing those labels with candidate password fragments.

Publish output safely

GNU shuf -o reads input before opening its output, allowing documented in-place shuffling forms. Shell redirection behaves differently: shuf input > input truncates the file before shuf reads it. Prefer a separate temporary output and a success-gated publication step when preserving the original population matters.

Check exit status, output record count, membership, and intended uniqueness or replacement policy before making a sample available to workers. A nonempty partial file is not proof that the command completed. Preserve the original and quarantined partial output on failure rather than mixing it with a later retry.

Test the contract and measure the workload

Include empty input, one record, a requested sample larger than the population, duplicate values, unterminated final text records, unusual filenames, a failed output write, and an exhausted random-source file. Assertions should verify invariants while allowing valid random variation.

Benchmark representative input size instead of assuming that every mode has constant memory or latency. Record sampling is a preparation stage whose cost can affect a test harness, especially if the input is streamed from a slow producer. Keep selection metrics separate from the measured application operation.

A useful randomized run has an explicit population, replacement rule, framing format, completion check, and replay artifact. shuf supplies the selection mechanism; those surrounding decisions make its output interpretable.

Related:

Sources: