tee in Shell Pipelines: Logging Without Hiding Broken Writers

tee copies standard input to standard output and to one or more files. It is useful when a pipeline must both feed a consumer and retain a log or artifact. It is not a transparent observer: it opens destinations, can truncate them, can fail partway through a write, and introduces another process whose status must be included in the pipeline contract.

The design question is whether each output is required. If a log is best-effort but the primary data path is mandatory, report those policies separately. If both outputs are required, any write failure must fail the job. A successful final consumer does not prove that tee wrote a complete log.

Choose replacement or append deliberately

By default, tee creates missing destinations and overwrites existing contents. Append mode changes that behavior. Be deliberate: replacing a file that is also an input can destroy data, and appending repeated CI runs can combine outputs from different versions without a clear boundary.

producer | tee run.log | consumer

The command above truncates run.log before writing on common implementations. If the pipeline fails after part of the stream is written, the log is partial. Add a run identifier, metadata header, and completion marker only if the consumer format permits them; otherwise record status in a separate manifest. The presence of a log file is not evidence of a complete run.

For a log that should not be exposed while incomplete, tee to a private temporary file, check every required status, then rename the finished log into the published location. Keep the temporary file on the same filesystem when atomic rename is part of the contract. File permissions matter: output may contain credentials, request bodies, environment diagnostics, or personal information.

Pipeline status depends on shell policy

In a POSIX shell, pipeline status normally comes from its last command. If tee fails writing a log but consumer succeeds, the overall status can still be zero. Bash pipefail makes a failed stage visible in an aggregate status:

set -o pipefail
producer | tee run.log | consumer

This does not identify which output failed. If per-stage diagnosis matters, capture Bash PIPESTATUS immediately after the foreground pipeline. Do not run another command first and then inspect it; the array describes only the most recent pipeline. Portable scripts can separate stages into temporary artifacts and check each status independently.

Broken pipes are another case. If consumer exits early, tee may receive SIGPIPE while writing to standard output or a destination. Some outputs may already contain a prefix and others may not. Treat early termination as intentional only when the consumer’s contract says it is safe. Do not blanket-ignore tee’s failure; doing so also hides disk-full and permission errors.

Extra consumers and process substitution

Bash, zsh, and ksh provide process-substitution forms that can send copies to extra commands. This is convenient but does not turn each side process into a normal pipeline stage whose status is automatically collected. A process substitution can fail while the main pipeline reports success. If an audit upload, checksum, or parser is mandatory, supervise it explicitly and wait for its status.

Avoid using a second copy for sensitive data without a redaction policy. A debug tee can leak authorization headers, cookies, database queries, or shell traces into build logs. Prefer structured redaction before fan-out, and test that secret-like fixtures do not appear in retained output. Redaction itself can fail or be incomplete; document which fields are masked.

Multiple files and partial writes

tee opens destination files and writes as the stream arrives. If one disk fills or a destination disappears, other destinations may already have received data. Multiple output files are not an atomic fan-out transaction. Use a staging directory, verify all artifacts, and publish a manifest or directory switch only after every write succeeds.

A file path supplied to tee is an operand, not a shell redirection. Quote it so spaces remain one path. Verify destination ownership and permissions before writing. If a destination could be a symlink in an attacker-controlled directory, tee’s path-based open is not a hardened no-follow protocol. Use a protected directory or a helper with descriptor-level protections.

When logging command output, distinguish stdout from stderr. tee sees only the stream connected to it. Redirecting stderr into stdout changes ordering and can mingle machine-readable output with diagnostics. If preserving separate streams matters, capture them independently and define how ordering is represented.

Validate completeness and protect logs

A useful log record has an invocation identity, tool versions, start and completion states, exit codes, and a digest or byte count for output where appropriate. Write completion metadata only after required consumers and outputs finish. An abrupt process kill may leave a partial file without an exit record; downstream readers should reject incomplete artifacts.

Test a healthy consumer, an early-closing consumer, an unwritable log directory, a full filesystem if safely simulatable, missing parent directory, and a producer that exits nonzero. Verify behavior under the target shell with and without pipefail. Confirm that a partial log is not mistaken for success and cleanup does not delete a previous known-good log before replacement is validated.

tee is a small utility with a large effect on data flow. Define overwrite versus append, required outputs, pipeline status, secret handling, and publication semantics. That turns observability into evidence rather than another way to hide failures.

Related:

Sources:

Choose replacement or append deliberately

Pipeline status depends on shell policy

Extra consumers and process substitution

Multiple files and partial writes

Validate completeness and protect logs

Comments