GNU dd Block Semantics: Short Reads, Counts, Padding, and Sync

dd is a byte-copying utility with configurable input and output block sizes. Its record counters look like a file format, but they report I/O blocks, not application records. The same command can behave differently when the input is a regular file, pipe, device, or short-reading producer. Before using dd in an image workflow, reason about its block contract, its output-file behavior, and whether a conversion changes bytes.

This guide covers GNU dd as documented by GNU Coreutils and tested here with GNU Coreutils 9.12. Do not assume the unprefixed dd on macOS or another Unix system accepts GNU extensions such as status=progress or conv=fdatasync. iflag=fullblock was added to POSIX.1-2024, but older or nonconforming utility versions may still lack it. Check the implementation and version with its own help or package inventory before using these examples operationally.

`ibs`, `obs`, and `bs` describe I/O blocks

ibs=SIZE controls the number of bytes requested for each input block; its default is 512. obs=SIZE controls output aggregation, also defaulting to 512. bs=SIZE sets both and overrides separate ibs and obs values. When bs is specified and no data-transforming conversions such as swab, block, or unblock apply, GNU dd writes each input block as it reads it. In a raw copy, an incomplete final block remains short rather than being padded.

For an ordinary file, this command copies four 4 KiB input blocks after skipping two such blocks:

dd if=source.img of=sample.bin bs=4096 skip=2 count=4 status=progress

The input position is 8,192 bytes, and the requested copy is at most 16,384 bytes. If the file ends early, the final input block may be shorter; dd does not invent bytes unless a conversion requests padding. Because of=sample.bin names a regular file, the default behavior truncates it before writing. Use a fresh disposable destination for inspection, not the only copy of an image.

count=4 here is four input blocks of 4,096 bytes, not four output writes, four application records, or necessarily 16,384 bytes in every circumstance. With separate input and output block sizes, count is still expressed in input blocks. GNU’s B suffix is an extension that changes a count to a byte count, but it is less portable than spelling the block size and count explicitly.

A read can return less than the requested input size

An input block size is a requested maximum, not a guarantee that a single read call will fill it. A pipe may provide fewer bytes than ibs even while more bytes will arrive later. Without iflag=fullblock, a short read can count toward count; a command intended to process two four-byte blocks can stop after receiving two small reads. GNU’s iflag=fullblock accumulates short reads until it has complete input blocks or reaches EOF, so count behaves as a count of complete input blocks.

This matters most for streams, not ordinary sequential reads from regular files. If a producer can be interrupted or emits data in chunks, include iflag=fullblock when a fixed number of complete blocks is the intent and the installed GNU version supports it. It cannot make a truncated source complete: at EOF, an incomplete final input block remains partial.

The phrase “records in” in dd statistics should not be interpreted as the producer’s record structure. For example, a line-oriented input has newlines that dd does not understand; a network producer may write one logical message over several writes. The counters describe full and partial dd blocks observed during this process. For protocol framing, use a parser that understands the protocol rather than treating the progress line as a record ledger.

Padding is a data conversion, not a durability option

conv=sync pads every input block to ibs with trailing NUL bytes. For block or unblock text conversions, GNU uses spaces instead. That can make a destination block-aligned, but it changes the byte stream and can turn an incomplete read into a full-size block containing synthetic data. It does not sync storage.

For example, with ibs=4, a final two-byte input block becomes four output bytes when conv=sync is selected. The default copy would preserve only the two available bytes. The choice depends on the target format. Do not add conv=sync merely to make records in and records out look balanced, especially when reading a disk image whose trailing bytes are meaningful.

conv=noerror asks GNU dd to continue after input read errors. Pairing it with conv=sync pads short or failed blocks, but does not create a map of unreadable ranges or the retry history provided by GNU ddrescue. A zero-filled sector can look syntactically plausible while replacing evidence of a read failure. For failing media, use an error-aware recovery tool and preserve its mapfile; do not treat dd conv=noerror,sync as an equivalent recovery system.

Output truncation and seek can destroy context

Unless conv=notrunc is used, GNU dd truncates the output file before copying. seek=N skips output blocks before writing, measured in obs-sized units. This can be useful for a deliberate write into a disposable image copy, but a misspelled of= or a mistaken block size can overwrite the wrong region or discard the rest of the output.

If an in-place offset update is truly required, create and verify a working copy first, calculate offsets in bytes, choose an explicit obs, and review both operands before running the command. conv=notrunc means “do not truncate the output file”; it does not mean “perform a safe patch,” “preserve a backup,” or “write atomically.” A failed write can still leave a partially modified file. The sample above avoids this risk by creating a new output path and reading an ordinary source file.

Treat device paths as high-risk operands. Reversing if= and of= can overwrite a source device. Do not paste examples that target /dev/disk*, /dev/rdisk*, /dev/sd*, or a mounted volume without confirming the device identity, unmount state, source direction, destination capacity, and backup plan. Prefer a regular file and a read-only inspection pass while learning the command.

Synchronization asks the operating system to flush output

GNU’s conv=fdatasync asks it to synchronize output data before finishing. conv=fsync asks to synchronize data and metadata. These are distinct from conv=sync, which pads input blocks. They are also different from the oflag=sync I/O flag, which requests synchronized output I/O rather than only a final flush.

A successful synchronization call is not a guarantee against every storage failure. Filesystems, device caches, controllers, virtual disks, and power-loss behavior affect what “persistent” means. For a release artifact, write to a staging file, check dd’s exit status, verify length and checksum, and only then publish it through the workflow’s atomic promotion mechanism. Do not append conv=fsync to a command and call an incomplete copy correct or authenticated.

GNU also supports status=progress to periodically update transfer statistics. Progress output is diagnostic, not a completion signal. Automation should check the process exit code, then validate the result. If the command runs in a pipeline, enable pipefail in Bash or capture each stage’s status explicitly; otherwise a succeeding consumer can hide a failed dd.

A safe review checklist

Before copying, write the intended byte offset and maximum byte count in human terms, then convert those numbers into skip, seek, ibs, obs, and count units. Confirm whether the source may short-read, whether any conversion changes content, whether the target is truncated by default, and whether the operation modifies only a disposable copy. Afterward, compare expected and actual output length and compute a digest where byte identity matters.

For an extraction from an image at offset 8,192 bytes, one useful mental check is: skip=2 with bs=4096 advances 8,192 bytes; count=4 requests four input blocks; the maximum result is 16,384 bytes. If the intended units were bytes rather than blocks, the command is wrong even if the transfer counters appear plausible. Add a fixture for an input shorter than one block, a final partial block, a pipe that deliberately delivers tiny writes, and an existing destination that verifies truncation or non-truncation behavior.

Keep simple copying separate from forensic recovery and storage policy. dd moves bytes according to its options. It does not know the image’s partition table, filesystems, bad-sector map, checksum, record framing, or safety consequences of a device path. Its flexibility is useful when each unit and side effect is explicit; it is dangerous when “block” is treated as an unambiguous synonym for byte count, record, or durable write.

Related:

Sources:

ibs, obs, and bs describe I/O blocks