yq in Shell Automation: YAML Types, Arguments, and Safe Updates

The name yq does not identify one universal program. Mike Farah’s Go implementation uses jq-like expressions and supports YAML, JSON, XML, INI, and other formats; a separate Python project named yq converts YAML to JSON and delegates filtering to jq. Their flags, expression languages, versioning, and output behavior are not interchangeable. A production script should identify the implementation, pin or constrain its version, and check it at startup rather than assume any executable named yq has the expected interface.

YAML is a typed data format, not a text file with indentation. A shell update should preserve that model: keep the expression as code, pass changing values through the tool’s data interface, validate the result, and publish it only after the transformation succeeds. Shell quoting protects an expression from the shell; it does not make the expression safe if external input is concatenated into it.

Pin the command and identify its version

Before running a transformation, determine which yq is installed. Mike Farah’s releases use a v4 command family, and yq –version provides implementation and version information in normal installations. A preflight should fail with an actionable message if the wrong implementation is present:

if ! command -v yq >/dev/null 2>&1; then
    printf '%s\n' 'required Mike Farah yq is not installed' >&2
    exit 127
fi
yq --version

For reproducible CI, install a pinned release from a trusted distribution channel and verify the artifact according to that channel’s provenance and checksum or signature process. Do not treat a version string alone as supply-chain verification. On a system where multiple package managers may install different tools under the same name, invoke the explicitly provisioned binary path.

Pass values as data, not generated expression source

Mike Farah yq supports environment-backed operators such as strenv(NAME) for passing a string value into an expression. This avoids injecting shell text into the program:

SERVICE_NAME='worker [blue]' yq '.services[] | select(.name == strenv(SERVICE_NAME))' deployment.yaml

The expression remains static and the variable is read as data. Use a correctly typed operator for the expected schema. strenv yields a string, even if contents resemble true, a number, or YAML syntax. If a boolean or integer is required, validate input and use an explicitly typed conversion supported by the pinned version. Avoid eval on an environment variable unless the variable is intentionally trusted expression code; eval interprets yq syntax and should not carry ordinary user input.

An update can use -i, but that changes a file directly and its semantics depend on implementation and version. For production configuration, write to staged output, parse and validate it, then replace the destination under a deliberate metadata and rollback policy. An in-place update that fails or emits an unexpected empty document may already have changed the live file.

YAML values are not interchangeable with strings

YAML has scalar typing rules. A value that looks like a number, boolean, null, date, or special scalar may be loaded with a type different from the human intent. Quoting can change the type. A release identifier such as 0017 should be tested as a string if leading zeros matter. Likewise, “false” is a string while false is a boolean. Validate type and value rather than merely checking whether a path exists.

YAML also supports multiple documents. Evaluation applies expressions to documents in sequence, so a script expecting one deployment object should assert document count or schema explicitly. Anchors, aliases, tags, comments, and formatting may be represented differently after parse and serialize. A semantic update can preserve data meaning while changing presentation; if comments or exact formatting are part of the review contract, test that behavior with the selected version and consider another editing strategy.

Validate schema and output cardinality

The -e option can change exit behavior when no matches occur or the result is null or false, but this does not replace schema validation. A filter may successfully return a value from the wrong document or silently update zero targets. Check required paths, types, and cross-field constraints. If exactly one object should change, assert target cardinality before and after.

For a deployment file, a safe process is:

Parse the original with the pinned yq implementation.
Validate input schema and target cardinality.
Generate staged output without replacing the original.
Parse staged output and validate the complete resulting schema.
Review a diff or assert that the intended field changed and unrelated fields did not.
Publish with the required ownership, mode, backup, and atomicity semantics.

The stages should have distinct failure messages. “yq returned zero” means only that the command completed according to its status contract; it does not prove a production configuration is valid or safe to deploy.

Shell data flow and secrets

Command-line arguments may be visible through process inspection and CI logs. Environment variables may also be exposed to child processes and diagnostics. Do not place secrets in an expression, print them through strenv, or include sensitive values in a diff. For secret-bearing configuration, use restricted file permissions, redact test output, and ensure temporary files are created in a private directory. A temporary file in a world-readable directory can leak contents before final replacement.

Be explicit about standard input and file operands. Mike Farah yq accepts a dash for standard input in applicable modes, but do not accidentally mix a stream of documents with a file list if the expression assumes one root object. Keep stderr diagnostics distinct from serialized stdout so downstream tools do not parse errors as YAML.

Test the exact implementation in CI

Create fixtures with a string that resembles a number, boolean, null, quoted delimiters, arrays, missing fields, multiple documents, anchors, comments, and empty input. Test a successful update, zero-match update, malformed YAML, invalid expression, wrong executable, and write failure. Assert parsed values and output semantics, not just a textual substring. Record yq –version in test logs and run the same pinned binary used in production.

The executable-name collision is a major operational hazard: verify which yq the package manager installed, and document the specific project and version family. Use a YAML-aware processor for YAML, but preserve the data/code boundary and add validation around it. That turns a terse command into a controlled transformation that can be reviewed, tested, and rolled back.

Related:

Sources: