Skip to content
Shell & TerminalDeep Dive Published Updated 7 min readViews unavailable

GNU Parallel Joblogs: Resume Work Without Replaying the Wrong Inputs

Choose GNU Parallel's resume modes, preserve sequence-to-input identity, and design retries around side effects without assuming exactly-once execution.

Parallel work is easy to start and surprisingly hard to resume safely. If a process stops halfway through a large batch, rerunning the same command can repeat completed side effects; changing the input list before restarting can assign an old completion record to the wrong item. GNU Parallel’s joblog and resume options provide a sequence-aware recovery mechanism, but they are not a transaction manager and they do not make arbitrary commands execute exactly once.

The central rule is to preserve the joblog’s sequence-to-input/argument mapping: keep the same ordered work list and joblog when resuming. GNU Parallel uses sequence numbers to decide which inputs remain, so reordering or replacing arguments can make old completion records apply to the wrong work. A deliberate command change can be valid for unrun or failed sequences when the argument mapping remains unchanged. This article covers GNU Parallel specifically; shell background jobs, wait, xargs, and GNU Make’s jobserver have different status and scheduling models.

What the joblog remembers

With --joblog FILE, GNU Parallel writes a tab-separated record for each job. The documented columns include sequence number, host, start time, runtime, transferred bytes, exit value, signal, and command. The sequence number is the crucial recovery key: the resume modes compare recorded sequence numbers with jobs generated by the current input and argument order. They do not discover whether an external system actually committed a side effect or whether a file contains valid output.

Create the log beside a stable input manifest and keep it as recovery state, not as disposable output. For example:

parallel --joblog run.tsv --halt soon,fail=1 \
  ./convert-item ::: item-001 item-002 item-003

If the command fails, preserve run.tsv, the exact ordered input list, the executable version, and the invocation. Do not recreate the input list from a directory glob that may now contain new files, remove failed rows by hand, or reuse the same log for a different batch. If inputs come from a file, archive or checksum that file with the log. Stable sequence numbers are only meaningful while sequence N still names the same work item.

The three restart modes answer different questions

--resume resumes from the last unfinished job and uses only sequence numbers: a sequence already present in the joblog is skipped regardless of its recorded Exitval or Signal. It therefore will not retry a failed row. GNU Parallel determines progress from sequence numbers, so preserve the input/argument sequence and joblog; changing a command for work not yet recorded can be deliberate, but changing the sequence-to-argument mapping can run the wrong work.

--resume-failed retries recorded failures and resumes from the last unfinished job. It treats a row as successful only when both Exitval and Signal are zero, and uses sequence numbers to match failures to the current input. The manual permits changing the command while preserving the input arguments and their sequence; that can be useful when repairing a failed operation. Do not reorder, insert, remove, or replace inputs, or substitute a different joblog, unless you have intentionally rebuilt and verified the sequence-to-argument mapping.

--retry-failed means “rerun failed jobs from the joblog.” Unlike --resume-failed, it takes both the command and arguments from the log and ignores command-line command text and arguments supplied for the retry. It is useful when the recorded command itself is what must be repeated, but it can also faithfully repeat an unsafe command. Inspect the saved command column before invoking it, especially if the log came from another user or machine.

These differences are easiest to see in a table:

Mode Checks Exitval and Signal? Work selected Command source
--resume No Skip recorded sequence numbers and resume from the last unfinished job Current invocation
--resume-failed Yes Retry failed rows, then resume from the last unfinished job Current invocation
--retry-failed Yes Failed rows recorded in the joblog Saved joblog command and arguments

Option behavior is part of the GNU Parallel version you run. Check the installed manual and parallel --version before relying on a recovery recipe copied from a different release.

Keep sequence identity stable across the restart

Suppose the first run uses three explicit IDs and a transient failure interrupts the batch. Resume from the same ordered IDs and the same program invocation:

parallel --resume-failed --joblog run.tsv --halt soon,fail=1 \
  ./convert-item ::: item-001 item-002 item-003

The restart is safe only if item-001 is still sequence 1, item-002 is still sequence 2, and so on. Do not insert a new item at the front, sort the list differently, or filter out a completed item and assume the old sequence numbers will follow the remaining names. GNU Parallel records execution state by sequence; it does not match jobs by a business key hidden inside your command.

When inputs are generated, materialize them first. Keep a plain manifest with one deliberately defined record per job, validate its order, and use that exact manifest on every run. If the intended work changes, start a new joblog for the new batch rather than trying to splice old completion state into a new sequence space. For a repair that changes command text, preserve the original manifest and inspect the log to confirm that only the desired failed sequences will receive the corrected operation.

Design the job for retry, not for wishful exactly-once execution

A joblog tracks process outcomes; it does not atomically commit an external side effect and its success record. A process can update a remote service or replace a file and then be interrupted before the outcome is safely recorded. A later retry may therefore encounter work that happened even though the restart policy considers it incomplete. Conversely, a recorded zero exit status only proves that the command reported success - not that the destination still contains the expected data.

Make each job idempotent where possible. Give each input a stable output name, write to a temporary file, validate it, and publish it with an atomic rename on the same filesystem. For remote changes, use an application-level idempotency key or a read-before-write reconciliation step if the service supports one. Keep retries bounded and inspect nonzero status, signal, and command columns before rerunning jobs that perform payments, destructive updates, or non-repeatable external actions.

Do not edit the joblog to “mark” a side effect complete unless you can prove the exact sequence, command, output, and success criterion. A manual row edit can conceal an incomplete or corrupted output just as easily as it can prevent a duplicate. Prefer a separate verified-results manifest if external reconciliation is part of the workflow, and document how it is joined to the GNU Parallel sequence list.

Validate recovery with a disposable batch

Before adopting a resume policy for production work, test it with a small fixture whose jobs leave separate, inspectable outputs. Force one job to return nonzero, allow other jobs to finish, and compare the joblog with the files actually produced. Confirm that --resume skips recorded failures, --resume-failed retries failures and continues unfinished work, and --retry-failed uses the command saved in the log. Then interrupt a job during its side effect and observe whether a retry repeats that side effect; this is the failure window that a successful normal rerun will not reveal.

Record the program version, exact command line, ordered input manifest hash, joblog path, and output verification result. Keep a copy of the initial log before experimental repair runs. Use --dry-run or a disposable destination to inspect commands before applying them to valuable data, but remember that a dry run cannot validate the external system’s commit semantics.

GNU Parallel’s joblog makes large command batches resumable when the sequence model is stable. Choose --resume when recorded sequences should be skipped regardless of their exit value, --resume-failed when failed work and the unfinished tail should run again, and --retry-failed when the saved failed commands are authoritative. The final safety property still comes from the job design: stable inputs, verified outputs, and retries whose side effects can be repeated or reconciled.

Related:

Sources:

Comments