Skip to content
Shell & TerminalDeep Dive Published Updated 7 min readViews unavailable

pv in Shell Pipelines: Progress, Back-Pressure, and Completion Evidence

Place pv deliberately in a pipeline, size progress estimates honestly, preserve exit status, and verify data integrity independently of transfer telemetry.

pv, the Pipe Viewer, shows how much data has crossed a point in a stream and can also modify transfer behavior. It is useful for a large image write, a compressed backup, an archive transfer, or a long import that would otherwise appear idle. The number it reports is a measurement at one location in a pipeline. It does not prove that a later process committed data, that a remote peer durably stored it, or that the resulting bytes match the source.

The project site lists pv 1.12.0, released September 17, 2026, as the current version at the time of writing. Distribution builds can lag, so check pv --version and the installed manual before relying on a recently added option. The fundamentals are byte counts, throughput, elapsed time, size-based progress, and stderr display. They are the right tools for most operational pipelines.

Put the meter where the question is

In producer | pv | consumer, the counter observes bytes crossing pv between the producer and consumer. pv can usually determine the size when it opens a regular-file operand directly, as in pv source.img; it cannot infer the size merely because an upstream producer is reading a file. If pv receives a pipe or another stream with unknown length, a percentage and ETA cannot be meaningful unless a total is supplied. The manual says an unknown-size progress bar indicates movement rather than completion percentage; --size supplies an assumed total for percentage and ETA calculations.

That assumed total is an estimate, not a validation. If the supplied size is too small or too large, the percentage and ETA mislead. --stop-at-size changes behavior further: when a size was specified, it stops transferring once that many bytes have been written, even if input remains. Do not add it casually to a recovery or backup pipeline. First determine whether the producer’s output is bounded and whether truncating excess input is actually the desired safety policy.

The meter’s position also determines what it cannot see. In pv source.img | dd of=target.img, the counter can show bytes handed onward from pv; it cannot certify that the block device accepted every byte or that its cache has flushed. A second meter after a compression command can show compressed output volume, not original input bytes. If the operational question concerns each stage, put named meters at deliberate boundaries and compare counts rather than assuming one bar describes the whole system.

Keep telemetry off the data stream

pv normally copies input to standard output and writes its visual display to standard error. It suppresses the display when stderr is not a terminal unless --force is used; forcing progress text into a file or log collector can make output noisy or unsuitable for machine parsing. Cursor-positioning output is specifically controlled by --cursor, so reserve that option for a terminal-aware display rather than ordinary CI logs. Keep the data channel and progress channel separate:

set -o pipefail

pv --name image --bytes --timer --rate source.img \
  | dd of=target.img bs=4M conv=fsync status=none

This is a GNU/Linux-oriented example: pipefail, GNU dd options, and device flush semantics vary by shell and operating system. dd of=target.img creates that path or truncates an existing file, so ensure the destination is distinct from the source and is the file you intend to overwrite. The command makes progress visible while sending bytes through the pipeline; it does not prove that target.img is a complete, correct copy. Verify size and content independently after the write, and ensure the source is a stable snapshot if it can change during the transfer.

Shells differ in how they report pipeline status. In Bash, set -o pipefail makes the pipeline return a failure if a non-final command fails, rather than only reporting the last command’s status. POSIX.1-2024 standardizes pipefail, but older POSIX editions and shells that do not yet implement Issue 8 may lack it; check the target shell and version. For a multi-stage production job, capture the statuses and report which component failed; a successful final compressor does not rescue a failed reader.

Progress and back-pressure are linked

Pipes are bounded queues. If the consumer is slower than the producer, it eventually fills the pipe and slows upstream writes; that is back-pressure, not necessarily a stalled pv. A displayed rate describes flow at the meter’s insertion point; its averaging window and refresh behavior depend on pv options. It can fall when a disk, network, compressor, checksum tool, or downstream process becomes the bottleneck. --rate-limit can intentionally cap a stream; --buffer-size and --pipe-buffer-size can change buffering on supported platforms, but a larger buffer does not guarantee higher end-to-end throughput.

The current manual documents that transfer buffer percentages and “last written” display require a userspace transfer buffer, so some options enable --no-splice. That can slightly reduce efficiency where splice(2) or copy_file_range(2) are available. Treat those flags as an explicit trade-off. Measure the full pipeline before changing buffers or disabling kernel-assisted transfer paths.

When the input is a protected command prompt or a slow producer that should not display a premature ETA, --wait delays output until the first byte is transferred. --delay-start waits for a configured time before showing the display. For CI logs, --numeric writes machine-friendly progress to stderr; be aware that its cadence and format are telemetry, not a stable artifact schema unless your script deliberately owns that interface. --name can label a meter, while --cursor controls terminal cursor placement and belongs only in a terminal-aware display; ordinary CI logs should avoid cursor-control escapes.

Integrity and recovery are separate contracts

A completed progress bar says pv believes it reached the size used for its display. It does not mean the output checksum matches. For a file-to-file copy, verify byte count and a cryptographic digest after successful pipeline exit:

set -euo pipefail
source_size=$(stat -c '%s' -- source.img)
pv --size "$source_size" source.img \
  | dd of=target.img bs=4M conv=fsync status=none

test "$source_size" = "$(stat -c '%s' -- target.img)"
source_hash=$(sha256sum -- source.img | awk '{print $1}')
target_hash=$(sha256sum -- target.img | awk '{print $1}')
if [[ "$source_hash" != "$target_hash" ]]; then
  printf 'checksum mismatch: source.img and target.img differ\n' >&2
  exit 1
fi

The stat syntax above is GNU-specific; use the target platform’s documented equivalent. Bash set -e can exit on an unhandled nonzero status, but its documented exceptions include tests in if, most commands in &&/|| lists, non-final pipeline stages unless pipefail applies, and commands whose status is inverted with !; it is not a substitute for deliberate error handling. Here, pipefail exposes a failed pv or dd stage, and the explicit hash comparison exits nonzero if the copied bytes differ. For databases or live virtual disks, use an application-consistent snapshot or quiesce writers first. A stream can be copied at high speed and still represent an internally inconsistent source if data changed during the read.

Use --skip-errors only when the recovery plan explicitly accepts damaged ranges. The manual says skipped read-error sections are emitted as null bytes. That preserves a stream shape under a chosen policy; it is not a repaired archive or a verified disk image. Preserve the original medium, log the skipped offsets and command version, and mark the output as degraded so no later process mistakes it for a clean backup.

Diagnose apparent stalls without losing evidence

When pv stops advancing, inspect both ends of the pipeline: is the producer blocked on input, is the consumer waiting for a device, is the destination full, or did an earlier command exit? Capture exit status, input and output sizes, and the exact pv options. A live rate of zero can mean a blocked consumer, a source that has not produced data, or a long period between writes; it is not enough to identify which one.

For repeatable troubleshooting, reproduce with a bounded test file, include pv --version, and compare throughput with and without optional transfer modifiers. Keep progress output on stderr so the bytes remain unaltered. If several meters are placed in a pipeline, use --name and terminal support such as --cursor only in an interactive display; plain CI logs should avoid cursor-control escapes unless the log viewer handles them.

pv is an observability and flow-control tool. Its strongest operational use is to make waiting visible, identify a slow stage, and enforce a deliberately chosen rate or size boundary. Integrity, durability, snapshot consistency, and correct shell failure propagation still belong to the surrounding pipeline.

Related:

Sources:

Comments