Skip to content
LinuxDeep Dive Published Updated 10 min readViews unavailable

Linux RWF_ATOMIC: Torn-Write Protection with pwritev2 and statx

Use Linux RWF_ATOMIC with statx and pwritev2 to request aligned torn-write protection, then separate it from durability and transactions.

Linux RWF_ATOMIC asks pwritev2() to issue one write with torn-write protection. When the requested operation is accepted and the storage stack honors its contract, a crash or power loss must leave the affected range as the complete old data or the complete new data, not a mixture. This is narrower than an application transaction: it covers one supported write range, not several records, directory changes, or a database commit protocol.

This interface is useful for fixed-size records such as a superblock, index slot, or small metadata page when replacing a file by rename is not the desired update model. It is not a CPU atomic instruction, a general guarantee for ordinary buffered writes, or a promise that any filesystem and disk can provide. Discover support for the specific file and validate the complete alignment contract before using it.

Do not confuse torn-write protection with rename atomicity

Three different guarantees are easy to collapse into the word “atomic”:

Mechanism Boundary it addresses
RWF_ATOMIC Whether one eligible write range can be observed after a crash as a torn mixture of old and new blocks.
Same-filesystem rename() Whether path lookup sees the old or new name binding, rather than a partially copied directory entry.
O_DSYNC, O_SYNC, RWF_DSYNC, or RWF_SYNC Synchronization of the write’s data and the metadata needed to retrieve it with the storage device.

An atomic write does not make a set of calls into a transaction. It does not update another file, synchronize a parent directory, validate a record’s checksum, or ensure that a controller with a volatile cache actually preserves data when power fails. Conversely, a crash-safe temporary-file-and-rename sequence can publish a complete pathname replacement without making an in-place multi-block write atomic. Select the primitive that matches the state transition, and document the rest of the commit protocol separately.

Probe the file instead of guessing from the kernel version

RWF_ATOMIC was added to the Linux pwritev2() interface in Linux 6.11. Availability for regular files is filesystem-specific. The current statx(2) documentation lists ext4 and XFS regular-file support since Linux 6.13. ext4 additionally requires its extents feature and an underlying block device that supports hardware atomic writes. Its single-filesystem-block mode requires a filesystem block size no larger than the system page size; the multi-filesystem-block mode instead requires bigalloc with an appropriately configured cluster size. Upstream XFS supports hardware REQ_ATOMIC offload and, on a reflink-enabled filesystem, a copy-on-write/remapping path. The latter is present in Linux 6.16 and can provide a filesystem correctness limit larger than the hardware-offload limit. A version check alone cannot establish that a particular file can use the operation.

Query statx() for STATX_WRITE_ATOMIC on the exact regular file. Check the returned stx_mask, not only the requested mask: it says which requested fields the kernel and filesystem actually supplied. Check that STATX_ATTR_WRITE_ATOMIC is present in stx_attributes_mask before interpreting the corresponding bit in stx_attributes. A supported file reports nonzero minimum and maximum atomic write units and a maximum vector-segment count. A missing result or zero unit means the application must not assume the feature is available.

For a preallocated fixed-slot update, also request STATX_TYPE and STATX_SIZE, verify both bits were returned, reject anything other than a regular file, and prove the entire slot lies within the reported file size before issuing the write. statx() is an observation, not a lock: prevent another owner from truncating or otherwise changing the file size between this check and the write/recovery sequence.

The returned stx_atomic_write_unit_min and stx_atomic_write_unit_max are byte limits for protected requests. stx_atomic_write_segments_max bounds the number of iovec entries. stx_atomic_write_unit_max_opt, when supplied, is a performance hint: it can be smaller than the correctness maximum, and a zero value means the maximum is the optimized limit. These are file-specific properties, not a value to cache globally for every mount or device.

RWF_ATOMIC also requires O_DIRECT. Query STATX_DIOALIGN and obey the file’s reported user-buffer, offset, and segment-length alignment as well. Do not substitute a hard-coded 512-byte or page-size assumption. For XFS, stx_atomic_write_unit_max describes the file’s correctness limit, while stx_atomic_write_unit_max_opt reports the hardware-offload limit when that separate limit is useful and available. Device capabilities, lower-layer alignment, and device atomic-write units constrain the hardware-offload path; they are not the sole basis for XFS’s filesystem-managed COW path. Use the exact file’s reported statx() limits instead of inferring support from a device specification.

Construct one compliant record update

The total byte count across the iovec entries must be a power of two, lie between the reported atomic-write minimum and maximum, and start at an offset naturally aligned to that total length. The number of vectors cannot exceed stx_atomic_write_segments_max. Direct-I/O alignment requirements apply at the same time, so an application must satisfy both sets of constraints. A single, correctly aligned vector is the simplest initial implementation.

The following is an illustrative C fragment, not a stand-alone compilable program: it omits headers, opening the test file, definitions for unit, slot_offset, and aligned_record, and the application-specific error/reconciliation helpers. A complete implementation must provide those pieces and open a pre-created regular test file with O_DIRECT, without O_APPEND, plus the synchronization policy it needs. This matters because Linux historically honored O_APPEND for pwritev2() and ignored the supplied offset; a fixed-slot write must not use an append-enabled open file description. Keep that descriptor’s shared status flags under application control while writing. Query the descriptor itself so the probe and subsequent write refer to the same inode:

struct statx stx = {0};
unsigned int want = STATX_WRITE_ATOMIC | STATX_DIOALIGN | STATX_TYPE | STATX_SIZE;
int status_flags = fcntl(fd, F_GETFL);

if (status_flags == -1)
    fail_closed("inspect open-file status flags");
if (!(status_flags & O_DIRECT))
    fail_closed("RWF_ATOMIC requires O_DIRECT");
if (status_flags & O_APPEND)
    fail_closed("fixed-slot atomic write requires O_APPEND disabled");

if (statx(fd, "", AT_EMPTY_PATH, want, &stx) == -1)
    fail_closed("statx atomic-write probe");

if ((stx.stx_mask & want) != want ||
    !(stx.stx_attributes_mask & STATX_ATTR_WRITE_ATOMIC) ||
    !(stx.stx_attributes & STATX_ATTR_WRITE_ATOMIC) ||
    stx.stx_atomic_write_unit_min == 0 ||
    stx.stx_atomic_write_unit_max == 0 ||
    stx.stx_atomic_write_segments_max < 1 ||
    stx.stx_dio_mem_align == 0 || stx.stx_dio_offset_align == 0)
    fail_closed("file does not expose the required contract");

if ((stx.stx_mode & S_IFMT) != S_IFREG)
    fail_closed("atomic fixed-slot target must be a regular file");
if ((uint64_t)unit > (uint64_t)SSIZE_MAX)
    fail_closed("atomic write length exceeds ssize_t");
if (slot_offset < 0 || (uint64_t)slot_offset > stx.stx_size ||
    (uint64_t)unit > stx.stx_size - (uint64_t)slot_offset)
    fail_closed("atomic write range is outside the preallocated file");

/* fd must remain free of O_APPEND; these values must pass every reported limit. */
struct iovec iov = { .iov_base = aligned_record, .iov_len = unit };
ssize_t n = pwritev2(fd, &iov, 1, slot_offset, RWF_ATOMIC);
if (n == -1)
    handle_atomic_write_error(errno);
else if (n != (ssize_t)unit)
    reconcile_partial_or_unexpected_result((size_t)n);

Compile a complete implementation against recent Linux UAPI and C library headers that define these interfaces. Before the call, verify that unit is a power of two within the reported min/max range and that the total iovec byte count is representable by ssize_t (for this one-vector example, unit <= SSIZE_MAX); pwritev2() returns EINVAL if the sum overflows ssize_t. Also verify that slot_offset is nonnegative and divisible by unit, the buffer address meets stx_dio_mem_align, the offset and length meet stx_dio_offset_align, and the vector count is permitted. The example also requests and checks STATX_TYPE/STATX_SIZE, rejects a non-regular target, and uses subtraction after an ordered bounds check to avoid overflow when proving the slot is within the preallocated file. fd should refer to that existing regular file on the tested filesystem, opened with the required direct and synchronized I/O flags and without O_APPEND; hold application-level exclusion against concurrent truncation while the probe, write, and recovery logic run, and do not allow another owner of its shared open-file description to enable append mode between the F_GETFL check and the write. In production, the fail_closed, handle_atomic_write_error, and reconcile_partial_or_unexpected_result example helpers need real error handling, metrics, and a recovery state machine.

The fixed record should be serialized to exactly the selected unit size, including initialized padding, and written wholly within a slot whose lifecycle the application owns. Keep a generation number and format validation inside the record when they help recovery, but do not treat them as substitutes for the kernel’s write contract. If the call fails, especially with EINVAL or EOPNOTSUPP, do not silently retry the same bytes through ordinary buffered I/O when the invariant depends on protection. A short or failed result must be handled as an uncertain state and reconciled before another writer overwrites the slot.

Keep durability and file metadata in scope

RWF_ATOMIC requests torn-write protection for the data range of that syscall; it is not itself a request to make the result durable. The pwritev2(2) contract requires O_SYNC or O_DSYNC on the open file description to synchronize the write, or the corresponding per-call RWF_SYNC or RWF_DSYNC behavior. The illustrative fragment does not show how fd was opened and passes only RWF_ATOMIC, so it does not itself request synchronized I/O. A complete implementation must select the required sync mechanism and check every write or synchronization result. Durable completion still depends on every layer honoring flushes. Hardware-offloaded atomicity additionally depends on every intervening block, device-mapper, RAID, virtual-disk, controller, and device layer truthfully preserving atomic-write requests; that device capability is not a prerequisite for XFS’s filesystem-managed COW path.

Keep the file size stable for a fixed-slot update. The statx() size check is not synchronization, so a concurrent truncate or extension can invalidate the checked range; serialize size-changing operations against the full fixed-slot update. RWF_ATOMIC does not turn an update that also changes file length or directory metadata into one all-or-nothing application transaction. If the application needs to publish a new pathname, update multiple records, or coordinate readers across generations, add a separate commit protocol such as a journal, version pointer, or synchronized rename. A successful protected write to one slot does not establish consistency of other state.

Validate on the actual storage path

Start with read-only statx() inspection on the exact test file, then test writes only on a disposable filesystem and data file. Record the kernel release, filesystem and mount, file type, reported masks and atomic units, direct-I/O alignments, device queue capabilities, and exact return values. Include negative probes for an unaligned offset, a non-power-of-two length, a length outside the advertised limits, too many vectors, and an unsupported filesystem. An expected rejection is useful evidence that the test reaches the relevant validation path, but it is not proof of power-loss behavior.

For an actual integrity test, alternate two recognizable record generations and verify the complete target slot after controlled crash testing on hardware and firmware representative of production. Killing the process or rebooting a virtual machine is not equivalent to cutting power to the storage device. Do not induce power loss on a production machine or on a disk with valuable data. A loop device may help exercise user/kernel argument validation, but it cannot prove that a physical controller and medium satisfy a hardware atomic-write guarantee.

Linux atomic writes fill a specific gap between buffered update protocols and storage-level torn writes. The safe design probes the target file, obeys both atomic-unit and direct-I/O constraints, asks for synchronized I/O when durability is required, and preserves an application recovery protocol for every state outside that single write.

Related:

Sources:

Comments