curl Downloads in Shell: Retries, HTTP Errors, and Atomic Files

curl can download a file successfully at the transport layer while the server returns an HTTP error page, write a partial file before a network failure, follow a redirect to an unexpected location, or retry an operation whose side effects have already occurred. A shell script that checks only “did curl run?” can therefore publish an error document as a release artifact or report a failed request as success.

Reliable download automation treats transport status, HTTP status, content validity, and publication as separate steps. Use HTTP-aware failure options, bounded retry policies, a private temporary file in the destination filesystem, an integrity check, and an atomic rename only after every check passes. The curl manual changes with releases, so verify option availability against the version installed on the target host.

Transport success is not HTTP success

By default, curl considers a completed HTTP exchange successful even when the response code is 404 or 500. This behavior is useful when a client needs to inspect response bodies and headers, but a shell downloader often wants an HTTP error to fail the command. --fail makes many HTTP responses of 400 or greater return a curl error. --fail-with-body also returns an error for those responses but retains the response body, which can be useful for diagnostics. It was added in curl 7.76.0, so check the version before using it in older enterprise images.

Pair failure behavior with visible diagnostics:

curl --fail-with-body --silent --show-error \
  --output response.json \
  'https://api.example.net/v1/status'

--silent suppresses the progress meter; --show-error restores error text when a transfer fails. The response body still needs validation. A successful 200 response can contain HTML from a proxy, a JSON application-level error, or an unexpected schema. Transport and HTTP success are necessary but not sufficient evidence that the artifact is usable.

Redirect behavior is another policy decision. --location follows redirects, which is commonly needed for signed download endpoints and object storage. Decide whether a redirect may change hosts, whether credentials may be forwarded, and what URL schemes are permitted. Do not use --location-trusted as a general fix for an authentication redirect; forwarding credentials to another host can leak them. Keep credentials out of URLs and shell history where possible, and use a scoped header file or an approved credential provider with strict permissions.

Retries need a bounded and repeatable operation

--retry N retries selected transient failures with a backoff strategy; it does not retry every HTTP failure by default. --retry-connrefused adds connection-refused errors. --retry-max-time bounds the period during which retries are attempted, while --max-time bounds an individual transfer. Choose both so one stalled attempt cannot exceed the job’s overall deadline.

For a read-only GET, a bounded example might be:

curl --fail-with-body --silent --show-error \
  --connect-timeout 5 --max-time 60 \
  --retry 4 --retry-connrefused --retry-max-time 150 \
  --output response.json \
  'https://downloads.example.net/releases/current.json'

The retry limit and total job deadline should be designed together. Curl’s retry timer determines whether it will begin another attempt; the final attempt can still run up to its configured transfer timeout. An outer scheduler deadline can terminate curl before its own reporting completes. Leave budget for verification and publication after the transfer succeeds.

Avoid enabling --retry-all-errors as a global default. Curl explicitly warns that retrying every error can have unintended consequences, including duplicate operations, and that retries cannot restore redirected output or input in every scenario. A retried POST or upload may have been processed by the server even if the client did not receive its response. For mutating requests, use an API-supported idempotency key, check the operation state, and define which status codes and transport errors are safe to retry.

Retry policy should distinguish 404 or authorization failure from a temporary server overload. Repeating an invalid token will not repair credentials; retrying a missing artifact can hide a deployment ordering defect. Use response codes, curl’s status, and server-provided retry guidance to decide whether to wait, fail, or escalate. Do not parse human-readable stderr as a stable API.

Never write a partial response directly over the published artifact

Download to a unique temporary file in the destination directory. If the transfer fails, validation rejects the content, or the process is interrupted, the old published file remains untouched:

#!/usr/bin/env bash
set -u

url=${1:?usage: download-artifact URL DESTINATION SHA256}
destination=${2:?usage: download-artifact URL DESTINATION SHA256}
expected_sha256=${3:?usage: download-artifact URL DESTINATION SHA256}
directory=${destination%/*}
[[ $directory != "$destination" ]] || directory=.
temporary=$(mktemp "$directory/.download.XXXXXX") || exit 1
cleanup() { rm -f -- "$temporary"; }
trap cleanup EXIT
trap 'exit 129' HUP
trap 'exit 130' INT
trap 'exit 143' TERM

if ! curl --fail-with-body --silent --show-error \
    --connect-timeout 5 --max-time 120 \
    --retry 4 --retry-connrefused --retry-max-time 240 \
    --output "$temporary" "$url"; then
  printf '%s\n' 'download failed; existing artifact was not replaced' >&2
  exit 1
fi

printf '%s  %s\n' "$expected_sha256" "$temporary" | sha256sum --check --status || {
  printf '%s\n' 'download checksum did not match' >&2
  exit 1
}

mv -f -- "$temporary" "$destination"

This example uses Bash and GNU sha256sum; it is not POSIX-portable as written. The expected checksum must come from a trusted, independently authenticated source. Downloading both the artifact and checksum from the same compromised endpoint does not provide independent integrity verification. Use a signature or transparency-backed release metadata where the supply-chain threat model requires authenticity, not only accidental-corruption detection.

mktemp creates a unique file with restrictive permissions on common implementations. Creating it in the destination directory normally keeps the final rename on one filesystem, where the name change is atomic to concurrent readers. It does not guarantee crash durability, preserve the old file’s mode or ownership, or prove that no other process modified the directory. Apply the intended metadata before publication, and use a protected destination directory for privileged artifacts.

The signal trap above removes the temporary file, but traps can interrupt shell logic in complex ways. For a production helper, test signals during both transfer and checksum validation. If cleanup itself fails, report that failure without masking the original curl status. Ensure the destination directory exists and is controlled by the intended user before starting the transfer.

Keep diagnostics separate from the downloaded bytes

Use --output to send the response body to a file and --write-out for selected transfer metadata. Do not interleave progress or status text with JSON consumed by a later program. For example, curl’s response code can be recorded in a separate variable or diagnostic log while the body remains in the staging file. Treat the body as untrusted input even when HTTPS succeeds; transport encryption authenticates the server endpoint according to the configured trust store, not the business meaning of the response.

Do not print authorization headers, cookies, signed URLs, or user credentials in verbose traces. --verbose is useful for a controlled troubleshooting session, but its output can include request headers and sensitive endpoint details. If verbose output is necessary in CI, redact it or use a disposable test credential. Disable shell tracing (set -x) around secret-bearing variables, and avoid passing tokens as command-line arguments when the platform can expose process arguments.

For HTTP APIs, keep response code, curl exit status, and application-level JSON fields separate. A response code of 200 with {"status":"failed"} may be a successful HTTP exchange and failed business operation. Conversely, a 202 Accepted response may mean the server queued work that has not completed. Poll a documented operation resource using a bounded policy rather than treating the initial download as proof of completion.

Do not confuse piping with preserving files or status

This command can hide a curl failure in a default POSIX shell:

curl --fail "$url" | jq '.release'

POSIX pipeline status is generally the status of the last command. If curl fails and jq accepts empty input or produces an output under the chosen filter, the script may miss the transfer error. Bash’s set -o pipefail changes the aggregate pipeline status, but it is Bash-specific and still gives only one status rather than a detailed per-stage report. For a release artifact, download to a file, check curl, verify the checksum or signature, then parse with jq as separate steps.

If a pipeline is appropriate for a streaming API, explicitly define how to preserve upstream and downstream failures. In Bash, capture PIPESTATUS immediately after the pipeline. In portable scripts, use temporary files or a carefully designed FIFO/wrapper protocol. A later printf or logging command overwrites $?, so save statuses before writing diagnostics.

Redirects, proxies, TLS, and protocol policy

Use HTTPS and keep certificate verification enabled. --insecure / -k disables peer verification and can turn an encrypted connection into an unauthenticated one; it is not a valid way to solve a broken certificate chain in production. Install the appropriate CA certificate or correct the server configuration. In a proxy environment, verify which proxy is trusted and avoid logging credentials embedded in proxy URLs.

Restrict allowed protocols if a URL can be influenced by outside data. Curl accepts many schemes, and some can read local files or interact with protocols beyond HTTPS depending on build support. Where relevant, options such as --proto and --proto-redir constrain the initial and redirected schemes. Validate hostnames against an allowlist before requesting a URL that comes from a manifest or user. Shell quoting stops shell metacharacters from splitting arguments; it does not stop SSRF, local-file access, redirect abuse, or an unexpected protocol.

Treat a URL’s path and query string as potentially secret. Signed URLs often grant access to an artifact and should not be retained in CI logs, shell history, or issue reports. For long-lived deployments, use short-lived credentials and an audited secret store rather than checking a token into a script.

Verify content before publication

For JSON, use jq after the download and enforce a schema predicate rather than merely running jq .. For a compressed archive, list or inspect it in a disposable directory and apply extraction safeguards before installing files. For a package, verify the publisher’s signature and package metadata. Check expected size bounds if unusually large or empty responses could indicate a server error. Use a content type only as a hint; headers are not a substitute for parsing and validation.

Use a trusted checksum or signature and bind it to the intended release version. Compare the final file’s checksum to the expected value, then rename it into place. If the target is a symlink-sensitive or privileged path, validate the directory ownership and use a secure deployment mechanism that cannot be redirected by an untrusted actor between checks. Atomic rename helps readers avoid partially written content but does not repair a compromised source or destination.

After publication, verify the artifact at its final path and record a release identifier, size, digest, and download result. If multiple processes can publish the same destination, add locking or compare-and-swap semantics. A successful mv is a filesystem operation, not a guarantee that downstream services have reloaded the file; add a separate health check where activation is required.

Test negative cases and pin assumptions

Test a timeout, connection refusal, DNS failure, TLS validation error, 404, 500, interrupted transfer, checksum mismatch, malformed JSON, redirect loop, and a successful but empty response. Confirm that every failed case leaves the previous published artifact intact and removes the temporary file. Use a local test server or disposable endpoint; do not provoke load or retry storms against a third-party production service.

Record curl --version in environment diagnostics so protocol and TLS backend differences are visible. Review the exact installed man page for --fail-with-body, retry defaults, redirects, and output behavior. Keep the endpoint, method, timeout, retry count, checksum source, and publication target in configuration with validation rather than constructing a curl command from a free-form string.

A dependable shell download is not just “curl with retries.” It is a bounded request with defined HTTP semantics, safe credential handling, distinct transport and application validation, and a publish step that runs only after content is trusted. Make each boundary observable and test its failure path before the artifact becomes part of a release or system update.

Related:

Sources: