jq in Shell Pipelines: JSON, Arguments, Streams, and Exit Status

jq is a JSON processor whose filter language runs over a stream of JSON values. In a shell pipeline, that makes it useful for validating API responses, extracting fields, constructing request bodies, and transforming event streams. The main reliability risks are not usually a complicated filter: they are mixing shell syntax with jq syntax, emitting raw strings into a second parser, assuming one input document or one output value, and treating every nonzero exit code as the same kind of failure.

The safe operating model is explicit: quote the jq program for the local shell, pass changing values through --arg or --argjson, decide whether the data is one document or a stream, and define what a successful jq result means to the caller. This article uses the jq 1.8 manual as its versioned reference. Check jq --version and the installed manual before relying on options that were added recently.

Shell quoting protects the filter, not the JSON input

On a Unix shell, put the jq filter in single quotes unless it must contain a shell-expanded value. The jq language uses $, parentheses, brackets, quotes, and pipes; many of those characters have their own meaning to the shell. The single quotes make the shell pass the filter as one argument without performing parameter expansion or command substitution inside it:

printf '%s\n' '{"service":{"name":"worker","enabled":true}}' |
  jq -r '.service.name'

The JSON is input data on standard input. The filter is code in an argument. Keeping those channels separate means a value such as $(touch /tmp/unexpected) remains a string from the JSON document; jq does not ask the shell to evaluate it. By contrast, constructing a filter string with shell interpolation can turn data into jq source and can also expose the shell to another parse if that output is later evaluated.

For a multi-line or reusable filter, store the program in a checked-in .jq file and use -f:

jq -f filters/active-services.jq inventory.json

That makes the transformation reviewable, syntax-checkable, and versionable. Do not place secrets in the filter file or command line. Command arguments can appear in process inspection, logs, or CI diagnostics, depending on the host and runner.

Pass data using jq’s argument options

If a shell variable is a string to compare against JSON, pass it as a value instead of inserting it into the filter:

expected_name='worker [blue]'
jq -e --arg expected "$expected_name" \
  '.service.name == $expected' config.json >/dev/null

--arg always binds a string, even if its text looks like a number, boolean, or JSON object. Use --argjson only when the variable already contains valid JSON text and should become a JSON value:

limit_json=25
jq --argjson limit "$limit_json" \
  '.jobs[] | select(.attempts < $limit)' queue.json

Validate that limit_json is intentionally JSON before passing it. If it is an ordinary external string, --arg is the correct choice; jq then performs type-aware comparisons rather than accepting arbitrary text as code. --args and --jsonargs provide positional values through $ARGS.positional, but use named arguments for filters where explicit roles help code review.

When output must be passed to another program, first decide whether the consumer expects JSON or plain text. Default jq output is JSON, so a string includes quotes and escapes. -r writes string results without JSON quoting. That is convenient for a controlled text field, but unsafe as a general way to serialize arbitrary data into another command: a newline in the string becomes a record boundary, and downstream tools may interpret shell metacharacters or leading hyphens. Use NUL-separated output with --raw-output0 only when every downstream tool preserves NUL and your jq version supports it. For structured interchange, keep JSON structured.

One input document and a stream are different contracts

By default, jq accepts a sequence of whitespace-separated JSON texts and runs the filter once for each value. A newline-delimited JSON file is commonly such a stream, but each line is not necessarily one object unless the producer guarantees that format. A normal pretty-printed JSON object spans multiple lines and is still one JSON value. Do not split JSON on lines with read and feed fragments back to jq.

The -s/--slurp option reads every input value into one array and runs the filter once. It is useful for small batches where global aggregation matters, but it consumes memory proportional to the full input. For example, jq -s 'map(.duration) | add' is appropriate only when the stream fits the machine’s memory and missing or nonnumeric fields have defined handling. Empty input, null, and malformed input require explicit policy rather than optimistic defaults.

For a large single JSON document, --stream exposes a stream of path/value records rather than materializing the entire tree in the normal way. A reduction can accumulate a small summary while parsing a large input:

jq --stream -n '
  reduce inputs as $entry
    ({records: 0};
     if ($entry | length) == 2 and ($entry[0][-1] == "id")
     then .records += 1
     else .
     end)
' large.json

Streaming is a lower-level interface, not a transparent memory switch. Empty arrays and objects have special path records, and the filter must reason about paths and container boundaries. Test a representative fixture containing empty objects, empty arrays, nulls, nested arrays, and malformed data if using --stream-errors. For most operational files, a clear non-streaming filter is easier to audit and plenty fast.

The --unbuffered option flushes each output as it is produced, which helps a live pipeline react to slow upstream data. It does not make the overall pipeline transactional or guarantee that the downstream process has persisted the data. Use an explicit transport and acknowledgement protocol if delivery confirmation matters.

Treat output multiplicity as part of the API

Many jq filters are generators. .items[] can emit zero, one, or many outputs for one input. A comma-separated filter can emit multiple results. A function may emit no result using empty. As a result, a shell script must not assume one input JSON object always produces exactly one output line. If the contract requires one JSON value per input, construct an array or assert cardinality deliberately.

For example, this emits one JSON array per input object:

jq '[.services[] | select(.enabled == true) | .name]' deploy.json

This emits one JSON string per selected service instead:

jq -r '.services[] | select(.enabled == true) | .name' deploy.json

The second form is easy to consume line-by-line only if names cannot contain embedded newlines and the consumer understands that a missing line may mean zero results rather than a processing error. A structured output avoids ambiguous records. Do not use head -n 1 to silently select the first result unless that is the documented rule; generators can produce values in an order that is not a meaningful business priority.

Make exit status semantics deliberate

Without -e, jq generally returns zero when the filter runs, even if the output value is false or null. With -e/--exit-status, jq returns status 0 when the last output value is neither false nor null, 1 when the last output is false or null, and 4 when no valid result is produced. Usage or system errors and filter compilation errors have other documented status values. This is not equivalent to “JSON parsed successfully” or “every record passed.”

If an assertion is the task, write a filter that yields a single boolean and use -e:

if jq -e '(.schema_version == 3) and (.services | type == "array")' config.json >/dev/null; then
  printf '%s\n' 'configuration accepted'
else
  printf '%s\n' 'configuration rejected or could not be evaluated' >&2
  exit 1
fi

If you need to distinguish false from parse failure or no results, capture the exit code and map jq’s documented status classes into your application’s own errors. Do not write jq -e '... | select(...)' and assume status 1 means invalid JSON: it can simply mean that the filter produced no matching output. Similarly, with multiple input texts, the documented -e result reflects the last output value; it is not an aggregate “all values true” check. Use all(...), any(...), or a deliberate reduction when the policy is about every value.

set -e does not repair unclear status semantics. In shell, a nonzero command may occur in conditional contexts where errexit does not terminate the script. Prefer explicit if checks around a jq validation step, and ensure downstream filters do not mask upstream failures. In Bash, set -o pipefail can make a pipeline report an upstream failure, but capture the status immediately if the script continues.

Write transformed files without exposing partial output

When jq generates a configuration file that another process will read, write to a temporary file in the destination directory, verify success, then rename it into place. A same-directory rename is normally atomic with respect to observers on one filesystem; it does not promise crash durability or preserve metadata automatically.

#!/bin/sh
set -u
input=${1:?usage: render-config INPUT OUTPUT}
output=${2:?usage: render-config INPUT OUTPUT}
directory=${output%/*}
[ "$directory" != "$output" ] || directory=.
tmp=$(mktemp "$directory/.render.XXXXXX") || exit 1
cleanup() { rm -f -- "$tmp"; }
trap cleanup EXIT
trap 'exit 129' HUP
trap 'exit 130' INT
trap 'exit 143' TERM

if jq -e '.schema_version == 3' "$input" >"$tmp"; then
  chmod 0644 "$tmp" || exit 1
  mv -f -- "$tmp" "$output" || exit 1
else
  printf '%s\n' 'input failed validation or transformation' >&2
  exit 1
fi

This example assumes a mktemp implementation that accepts the template form shown and a mv that supports --; check platform requirements if claiming strict POSIX portability. For untrusted destinations, also validate the directory and permissions. If mode, owner, ACLs, or extended attributes are part of the artifact contract, set and verify them intentionally before the rename.

The code deliberately validates with -e, but a filter that returns true for an invalid schema can still accept it. Replace the predicate with the actual schema policy, including required fields, types, ranges, and unknown-field rules where they matter. JSON syntax validation and business validation are separate checks.

Keep HTTP failures and JSON failures distinct

A common pipeline is curl URL | jq FILTER. By default, curl can successfully download an HTTP error response and still exit zero if the transfer itself succeeded. jq can then report a JSON parse error, or worse, parse a valid JSON error body and make the pipeline look like a successful API response. For HTTP-aware validation, configure curl’s failure behavior and check each stage deliberately:

if curl --fail-with-body --silent --show-error "$endpoint" |
   jq -e '.status == "ready"' >/dev/null; then
  printf '%s\n' 'service is ready'
else
  printf '%s\n' 'request, JSON parsing, or readiness check failed' >&2
  exit 1
fi

This POSIX pipeline still returns the status of the last command by default. If curl fails but jq receives empty input and returns a result under a permissive filter, the curl failure can be hidden. On Bash, enable pipefail or capture the response to a temporary file, check curl’s status, and then run jq separately. For reliable diagnosis, separate transport status, HTTP status, JSON syntax, schema validation, and application readiness into named checks. Avoid logging bearer tokens or unredacted response payloads in CI.

Test filters as programs, not as pasted one-liners

Keep important filters in files and use jq’s --run-tests format or a test harness with golden inputs and outputs. Include valid, empty, malformed, wrong-type, multiple-document, and adversarial string fixtures. Assert both stdout and exit status. A test that checks only the pretty-printed happy path misses null handling, generator multiplicity, incorrect raw output, and failure masking.

Pin the jq version in CI when behavior depends on a specific feature or numeric handling. Print jq --version in diagnostic logs, but not request payloads. Verify target hosts actually install jq and that its binary is trusted; do not download an unverified executable as part of a deployment. If an input is too large for regular parsing, benchmark --stream against the real data and budget memory for the reduction state and downstream consumer too.

Finally, keep the code/data distinction visible in review. The filter should be a constant string or checked-in file; values should cross through --arg, --argjson, input files, or standard input. Output should have an explicit shape and encoding. Exit status should express the predicate the caller cares about. Once these decisions are part of the script’s contract, jq becomes a predictable JSON tool rather than another layer of quoting folklore.

Related:

Sources: