Fixing Locale and Encoding Failures in Shell Tools

When a shell tool rejects text as an invalid character sequence, displays mojibake, or sorts the same file differently on two systems, first separate three things: the file’s bytes, the process locale, and the terminal’s rendering.

Inspect before converting

locale
file --mime sample.txt
hexdump -C sample.txt | head

file makes a heuristic guess, not a guarantee. A hex dump reveals byte-order marks, NUL bytes, or clearly mixed encodings. Preserve the original before trying any conversion; repeatedly transcoding already-damaged text makes recovery harder.

Understand locale precedence

LC_ALL overrides every other locale category. Category variables such as LC_CTYPE and LC_COLLATE override LANG. A UTF-8 locale tells many programs how to interpret multibyte text and classify characters, while the C locale asks for byte-oriented, reproducible behavior.

LC_ALL=C sort identifiers.txt
LC_ALL=C.UTF-8 python3 process_names.py

Use C deliberately for byte-stable pipelines, not as a universal Unicode repair. The exact available locale names vary; check with locale -a rather than assuming en_US.UTF-8 exists.

Convert only with a known source encoding

iconv -f WINDOWS-1252 -t UTF-8 legacy.txt >legacy.utf8.txt

The source label must match reality. If a file contains a mixture of encodings, one global iconv command cannot infer which bytes belong to which character set. Validate the converted file and compare record counts before replacing anything.

Check the display layer last

A valid UTF-8 file can still show boxes when the selected font lacks glyphs, and a terminal can be configured for a different encoding than the process expects. Test output by redirecting it to a file and inspecting bytes. If the file is correct but the screen is wrong, focus on the terminal, font, SSH locale forwarding, and $TERM path rather than rewriting the data.

For automation, set the required locale explicitly at the job boundary and document it. Depending on whatever locale happens to be inherited from an interactive desktop makes parsing and collation non-reproducible.

Where UTF-8’s specific bit-packing scheme actually came from

UTF-8 wasn’t designed in a standards committee over months of deliberation — Ken Thompson sketched its actual bit-packing scheme on a placemat at a New Jersey diner in September 1992, working through the problem with Rob Pike after Bell Labs got a call from an X/Open committee looking for a byte-oriented, ASCII-backward-compatible alternative to the wider, fixed-width Unicode encoding Plan 9 had been using and disliked. Pike and Thompson implemented it within days and it was accepted by X/Open shortly after — the encoding’s genuinely clever property, self-synchronizing byte sequences that never produce a valid ASCII byte as part of a multibyte character, traces directly back to that single evening’s design session, which is part of why UTF-8 remains so mechanically simple to reason about compared to alternatives designed by larger committees.

When `locale -a` doesn’t show the locale you actually need

On Debian and Ubuntu-family systems, a locale can be genuinely supported but not yet generated on your specific machine — locale -a only lists locales that have actually been built, not every locale the system theoretically supports. Adding the needed line (for example, en_US.UTF-8 UTF-8) to /etc/locale.gen and then running locale-gen as root builds it and makes it available; dpkg-reconfigure locales offers the same operation through an interactive menu instead of hand-editing the file directly. This is worth checking specifically before concluding a locale-related tool failure is a deeper bug — a surprising number of “unicode doesn’t work” reports trace back to a UTF-8 locale that was simply never generated on that particular system at all. Related: How Shell Expansion and Globbing Actually Work · The History of the Unix Shell: From Thompson Shell to Bash and Zsh

Sources: