Rsync Partial Transfers: Safe Resume, Staging, and Recovery Contracts

Rsync can restart an interrupted transfer efficiently, but “resume” is not one mode with one safety guarantee. The default receiver removes a partially transferred file after interruption. --partial keeps an incomplete file, while --partial-dir stores partial data separately and can use it as a basis on the next run. --inplace, --append, and --delay-updates change different parts of the recovery and publication contract. Choose among them based on whether a destination file may be visible while incomplete, whether it is truly append-only, and how much temporary disk space is available.

This guide covers rsync’s file-level transfer behavior. It does not make a multi-file deployment transactional, guarantee crash-durable storage, or replace a manifest-based backup system. Test both rsync endpoints and the destination filesystem before relying on a recovery option in production.

Understand what remains after interruption

By default, rsync deletes a partially transferred file when a transfer is interrupted. --partial changes that behavior to keep the partial file, which may make a subsequent run faster. It is useful for a large file that is expensive to resend, but it is not a general promise that a consumer will never observe incomplete content. Once the transfer has failed, treat the path as untrusted until a successful rerun and the application’s own validation complete.

For shared destinations, --partial-dir is usually a clearer boundary. It implies partial retention but stores the partial file in a designated directory instead of replacing the destination file with that partial result. On the next transfer, rsync can use a saved file from that directory to speed up resumption and removes it after it has served its purpose. A relative directory such as .rsync-partial is relative to the destination hierarchy; rsync adds a perishable exclude rule so that partial files are not accidentally sent back or removed by a later transfer.

rsync -a --partial-dir=.rsync-partial --info=progress2 \
  ./release/ [email protected]:/srv/site-staging/

The example writes to a staging tree that is not yet serving users. Keep the partial directory private: the rsync manual explicitly warns that it must not be writable by other users and calls out shared locations such as /tmp as unsafe. If custom filters are in use, review their order; they can override or make ineffective the automatic partial-dir exclusion. Do not add --delete casually to a recovery command. When stale partial data must be cleaned, follow the documented filter and delete timing so rsync can use valid partial data during the current run before cleanup.

Rsync creates only the final missing component of the partial directory path, not every parent component. Pre-create required parent paths with deliberate ownership and permissions. When both sender and receiver are at least version 3.2.0, the manual documents that resuming a file from a partial directory updates that partial in place rather than creating yet another full temporary copy. This affects peak receiver disk usage; it does not mean the final destination is globally atomic or that both peers can ignore version compatibility.

Keep staging separate from publication

The ordinary update path creates a new copy and moves it into place after the file is complete. A partial directory helps preserve an existing destination file while an interrupted new copy is waiting for a retry. For a deployment, point rsync at a versioned staging directory, validate the result, then switch the application to the verified release using a separately designed publication mechanism. Keep the previous release available until health checks pass.

--delay-updates can put updated files in a holding area until the end of the transfer and then rename them into place in rapid succession. This makes a set of file replacements closer to atomic, but it is not an atomic transaction across all files: readers can observe a mixture during the succession of renames, and a crash can still leave operational recovery work. The option needs additional disk space for updated files, uses more receiver memory, and conflicts with --inplace and --append. Do not describe it as a directory-wide snapshot.

--inplace is a different tradeoff. It writes changed bytes directly into the destination file instead of creating a complete new copy and moving it into place. During a transfer, the destination content is inconsistent; interruption can leave it that way. Existing hard links continue to refer to the changing inode, and other readers or running programs can observe damaging intermediate data. Rsync documents that --inplace conflicts with --partial-dir and --delay-updates; it is not a switch to combine with every “resume” flag. Use it only when those effects are intentional and the application can tolerate them.

Reserve append modes for genuinely growing files

--append and --append-verify are specialized modes for files known to be growing, where the receiver’s existing bytes are already the same prefix as the sender’s. --append writes only the appended portion and does not verify the old content; it can be dangerous if the files are not shared growing files. --append-verify includes existing data in checksum verification, which is safer in that specific use case but less efficient. Both are inappropriate as generic retry options for mutable configuration, database snapshots, or arbitrary release artifacts.

Before using an append mode, establish the producer’s write contract: the file is append-only, the receiver cannot contain divergent bytes, the source will not shrink or be rewritten in place, and no consumer depends on an intermediate append. If any assumption is false, use normal rsync delta transfer with a partial directory, or transfer into a fresh staging tree and validate it.

Handle filters, deletion, and receiver-side space

Relative partial directories receive an automatic perishable exclude rule, but local filter rules can change the outcome. The upstream manual describes a cleanup approach using --delete-after plus a risk filter rule to allow old partial directories to be removed after they can be used. That is a deliberate policy: --delete-before or --delete-during can remove a saved basis before the current transfer has a chance to benefit from it. Run a dry run against representative paths and inspect the exact filter list before enabling deletion.

Budget destination storage for more than final file size. Depending on the selected algorithm and timing, the receiver may need the existing destination, a saved partial, and a temporary output. --delay-updates can require another copy of all updated files. When the filesystem cannot fit the staging data, a failed recovery may be caused by storage pressure rather than network instability. Alert on free space and partial-directory growth; define a reviewed cleanup procedure instead of removing hidden directories indiscriminately.

Diagnose and verify a recovery run

First record both peers’ rsync --version, the exact command, exit status, and the destination filesystem’s available capacity. Then check whether the destination’s filter rules protect the partial directory, whether the source changed since the failed run, and whether an implied --whole-file option causes partial-dir files to be discarded. A partial basis is an optimization, not an authoritative checkpoint: rsync still compares and transfers according to its normal file-update algorithm.

For a high-value artifact, verify a trusted checksum or signed manifest after the successful transfer. Rsync’s successful exit status reports the transfer operation, but a deployment system should independently confirm artifact identity, ownership, expected file set, permissions, and application health before switching traffic. Keep transfer status distinct from publication status in automation: a rerun can recover bytes correctly while a later verification or release step still fails.

Exercise the recovery path with a disposable large file: interrupt once with default behavior, once with --partial, and once with --partial-dir; confirm where the bytes remain and whether the next run uses them. Repeat with custom excludes, a full destination filesystem, a source file that changes during transfer, and a receiver running a version older than 3.2.0. The test should demonstrate not just speed, but what readers can see and what cleanup is safe.

Related:

• Rsync Remote Arguments: Preserve Paths Across SSH
• curl Downloads in Shell: Retries, HTTP Errors, and Atomic Files

Sources:

• RsyncProject upstream rsync.1.md manual (reviewed commit)
• RsyncProject official source repository