sync in Shell Scripts: Writeback Is Not a Transaction or a Backup

The sync utility asks the operating system to synchronize buffered file or filesystem changes with persistent storage. It is relevant when administrators flush writes before maintenance or when scripts handle removable media. It is often misunderstood as a universal “make my data safe now” command. sync does not make a group of writes transactional, does not repair a broken application write protocol, does not verify the media after writing, and does not replace a database’s documented durability interface.

The operating system caches writes because storage devices are much slower than memory. A successful write can mean that data reached a kernel buffer, not that every layer below it committed the bytes to nonvolatile media. Filesystems, device drivers, controller caches, virtual disks, network storage, and hardware all participate. The guarantee depends on the system calls, mount configuration, device behavior, and whether the storage stack honors flush requests.

What the utility actually requests

The portable sync utility requests that pending filesystem changes be synchronized. On GNU systems, invoking sync without file operands synchronizes system-wide buffered state. GNU Coreutils documents file operands as using file-specific synchronization by default and offers implementation-specific modes for data-only or filesystem-wide synchronization. These options are not interchangeable and are not uniformly available across platforms; check the target manual before using them.

A global sync can affect unrelated workloads and may take time when the machine has substantial dirty data or a slow device. It is not a performance measurement technique and should not be inserted into a hot loop. A file-specific operation narrows the target but has an important limit: shell syntax cannot express every required ordering step for creating, renaming, and persisting a pathname. Applications that need crash-consistent publication generally need to synchronize file contents and then, where required by their OS/filesystem contract, the containing directory after a rename.

This distinction matters for the common temporary-file publication pattern. Writing a temporary file, checking its contents, and renaming it can prevent concurrent readers from observing a partial file under same-filesystem rename semantics. That is a namespace property. It does not necessarily prove that the data and directory update survive sudden power loss in the desired order. GNU mv and sync are utilities, not a substitute for the application-specific file and directory synchronization sequence documented by a database or storage API.

Durability, ordering, and the storage stack

Durability is an end-to-end property. A filesystem may acknowledge a synchronization request after issuing flushes, while a drive with a volatile write cache may still need correct firmware and power-loss protection to honor them. A virtual machine may pass the request to a hypervisor, which has its own cache policy. A network filesystem can acknowledge data according to server policy. A container shares the host kernel’s underlying storage semantics. Do not claim power-failure durability based solely on a successful exit status from a shell command.

Ordering also matters. If an application writes a data file and then writes a separate manifest saying “complete,” the manifest must not become durable before the data it describes if recovery relies on that order. sync across the whole machine does not express the application’s dependency graph or atomic commit point. Databases solve these problems with logs, checksums, ordering barriers, and recovery procedures. For custom shell automation, keep the protocol simple and use a storage API that can express required guarantees rather than improvising a transaction from commands.

Safe operational use

For removable media, the responsible workflow is to finish copying, check every producer and copy status, verify a digest or file inventory, request synchronization, and then use the platform’s supported safe-eject operation. A successful sync does not prove that all files match the source; verification is a separate step. Do not unplug a device merely because the command returned quickly, especially if the filesystem is remote or the desktop environment maintains its own mount state.

If a script invokes GNU file-specific sync, make its implementation scope explicit:

sync -- "$staged_file"

This requests synchronization for one file with GNU Coreutils semantics; the -- option and operand behavior must be checked on the target. A portable script should not assume that every operating system accepts a pathname operand or provides GNU’s data-only and filesystem-wide flags. Capture the exit status and report the exact target. A failure means the caller cannot assert the requested operation completed; it does not necessarily mean no bytes reached storage.

Avoid running a broad sync automatically after every shell write. It can introduce latency and hide a missing application-level commit design. Use it only when the operational requirement and target platform are clear, and measure the effect under realistic I/O load. If a tool generates a critical artifact, its strongest practical guarantee often comes from writing to a temporary file, validating it, publishing it with the documented same-filesystem operation, and having the consumer verify the artifact before use.

Test the failure model, not just the command

Test on the actual filesystem and storage configuration. Include disk-full behavior, read-only remounts, device disconnects, network-storage interruption, a process killed during write, and system restart at controlled points. Confirm what data is present after recovery and whether the application can distinguish a complete version from a partial one. A VM power-cut test may not faithfully model hardware caches, but it is still better than treating one successful command invocation as proof.

Record filesystem type, mount options, device or hypervisor, utility implementation, and the intended failure model. For safety-critical or financial state, rely on the database or application vendor’s durability guidance and independently verify recovery. A shell can orchestrate checks and surface errors, but it cannot turn a non-transactional storage operation into a transaction with a one-line sync call.

Related:

Sources:

What the utility actually requests

Durability, ordering, and the storage stack

Safe operational use

Test the failure model, not just the command

Comments