uniq in Shell Pipelines: Adjacent Groups, Counts, and Identity Policies

uniq compares adjacent records and groups equal neighbors. Its name can suggest global duplicate removal, but it does not retain a set of every record seen earlier. Whether it reports all repeated values depends on the order supplied by the producer and on the comparison rule selected by the options.

That behavior is useful in its own right: an ordered event log can be compressed into runs without destroying sequence information. Sorting first changes the question to grouped frequency by value, and it discards the original chronology. Choose deliberately between those analyses instead of automatically inserting sort whenever uniq produces unexpected output.

This article uses GNU Coreutils and Bash. Basic adjacent grouping is portable, while GNU options such as --group, --all-repeated, and NUL termination need implementation checks on other systems. Keep a stable locale and a documented record format for machine-readable reports.

Read counts as group sizes

For input up, up, down, up, the final up is a new group because it is separated from the first two. uniq -c therefore describes three runs, not two globally distinct states. That is useful for analyzing state transitions, but unsuitable as an overall event-frequency count without grouping the entire population first.

#!/usr/bin/env bash
set -euo pipefail
export LC_ALL=C
work=$(mktemp -d)

printf '%s\n' up up down up > "$work/events"
uniq -c "$work/events" > "$work/runs"
sort "$work/events" | uniq -c > "$work/frequencies"
cat "$work/runs"
cat "$work/frequencies"

The run sizes are two, one, and one in original order. The frequency report contains one down and three up occurrences in sorted order. These are expected fixture results; the padded human-readable count columns are not a stable CSV schema.

If the report needs timestamps or other fields, define which parts determine equality. A whole-line comparison sees two otherwise identical events with different timestamps as different records. Stripping timestamps might support one analysis but erase the information needed for another. Preserve the original input before normalizing it.

Unique and repeated have specific meanings

Default uniq retains one representative of every adjacent group. -d prints one representative only for groups that repeat. -u prints groups whose size is one. Consequently, uniq -u does not mean “deduplicate everything”: it removes repeated groups entirely.

printf '%s\n' apple apple pear plum plum > "$work/fruit"
uniq -d "$work/fruit" > "$work/repeated-groups"
uniq -u "$work/fruit" > "$work/singleton-groups"
printf '%s\n' apple plum > "$work/expected-repeated"
printf '%s\n' pear > "$work/expected-singletons"
cmp "$work/repeated-groups" "$work/expected-repeated"
cmp "$work/singleton-groups" "$work/expected-singletons"

If the same fruit appears again after a different value, it forms another group. To find globally repeated full records, sort them under the same intended equivalence before running uniq -d. To find duplicate entity IDs, extract or compare those IDs instead; full-record equality and key equality are different invariants.

Sort for the comparison you actually use

Locale affects comparison. Set LC_ALL=C consistently for byte-oriented deterministic analysis rather than sorting under one locale and grouping under another. If case-insensitive grouping is intended, establish matching sorting rules and inspect how folding affects identity.

Options -f, -s, and -w can skip leading fields, skip characters after those fields, or limit how much of the remaining content is compared. Their field model is blank-based, not arbitrary CSV or pipe-delimited fields. Applying -f 1 to a quoted CSV row does not skip the first CSV column.

Skipped comparison fields remain in the output representative. If two lines with different timestamps compare equal after skipping a timestamp field, default uniq retains the first representative in that group. That is not an aggregation that merges their metadata or chooses the newest timestamp. If the application needs a canonical row, specify its selection rule separately.

In addition, short lines can compare as an empty suffix after skipped fields or characters. Malformed input can therefore fall into a single group instead of causing a parser error. Validate required fields and minimum structure before interpreting grouped output as clean data.

Preserve duplicate evidence instead of hiding it

GNU --all-repeated retains every line from repeated groups, while -d retains only one. The former is useful when investigating which source rows produced duplicate identifiers. --group can emit every group with explicit separators for human inspection.

Those display delimiters can become ambiguous when the input itself contains empty records. A blank line separating groups is indistinguishable from a blank input line without another encoding rule. Do not remove meaningful empty records merely to make the display easier to parse. Keep a machine-readable original and generate a separate presentation report, or use a structured format with explicit counts and group boundaries.

For NUL-delimited data, GNU -z changes record framing and related grouping delimiters. Maintain that framing through the consumer. Counting newline characters cannot count NUL-separated filenames, and a newline within one filename must not split it into several apparent records.

Global deduplication changes order and multiplicity

sort -u is often the simpler tool when only a globally distinct list is needed. However, it changes ordering, and its equality can follow selected sort keys rather than a whole-row policy. Removing duplicates by key may discard rows whose other fields differ. Verify the exact sort comparison instead of assuming that -u means the same thing in every invocation.

For an ordered log where the first occurrence must survive, a sort-based pipeline does not preserve that original order automatically. Use an appropriate keyed algorithm and retain sequence numbers if that is the requirement. For very large inputs, also account for sorting’s temporary-storage requirements and failure behavior.

When duplicates represent a broken uniqueness promise, deleting them is not a repair. Produce a diagnostic report, identify the upstream cause, and decide how conflicting values should be reconciled. A pipeline that quietly suppresses one of two inconsistent account records may look cleaner while returning incorrect information.

Publish only after complete checks

Write analysis to a new output file. Avoid redirecting into the input file, which truncates it before processing. A successful grouping operation can still analyze the wrong input, so verify the source identity and expected record count alongside exit status.

Useful invariants include total input occurrences equaling the sum of group counts, singleton plus repeated-group membership covering the population under the selected grouping, and diagnostic duplicate IDs matching the schema rule. For a chronological run report, verify that concatenating the expanded runs recreates the original sequence.

Test separated duplicates, all-unique input, all-equal input, empty records, an empty file, case variants, malformed short rows, multibyte text under the chosen locale, NUL framing, and failed output writes. Distinguish utility behavior from application-specific validation: the command is not a schema checker.

uniq is most reliable when treated as an adjacent-group operator. Make ordering, equality, representative selection, and output framing explicit, and its counts become reproducible evidence rather than a plausible-looking list of “unique” records.

Related:

Sources: