How to Run Parallel Shell Jobs and Collect Every Exit Status
A bounded shell concurrency pattern covering background PIDs, wait status, failure aggregation, cleanup, signal forwarding, logs, and POSIX versus Bash features.
Appending & starts a command asynchronously in the shell’s environment and $! captures its process ID. The parent must retain each PID and call wait; otherwise it can report success before work finishes or lose individual failure statuses. Correct parallelism also needs a concurrency limit, signal policy, isolated outputs, and deterministic aggregation.
Capture PID and job identity together
For a small POSIX-compatible fixed batch, store explicit variables or a newline-delimited mapping that cannot be confused by filenames. Arrays are not POSIX. This three-job example makes the status logic clear:
run_one alpha >alpha.log 2>&1 & p1=$!
run_one beta >beta.log 2>&1 & p2=$!
run_one gamma >gamma.log 2>&1 & p3=$!
failed=0
wait "$p1" || { s=$?; printf 'alpha failed: %s\n' "$s" >&2; failed=1; }
wait "$p2" || { s=$?; printf 'beta failed: %s\n' "$s" >&2; failed=1; }
wait "$p3" || { s=$?; printf 'gamma failed: %s\n' "$s" >&2; failed=1; }
exit "$failed"Capture $! immediately; another background launch overwrites it. Capture $? immediately inside the failure branch. Waiting in launch order does not serialize execution—the jobs already run concurrently—but a later failure may not be reported until earlier jobs finish.
POSIX wait PID returns the asynchronous command’s status, subject to the shell’s retained-status rules. Waiting twice may not reproduce it. A subshell PID can represent a compound workflow, which is useful for grouping setup, action, and per-job cleanup under one status.
Bound concurrency
Launching one process per input can exhaust descriptors, memory, processes, network connections, or a remote service. For portable bulk work, xargs parallel options are common but not fully uniform across every POSIX environment. A worker-pool design with named pipes is possible and complex. If Bash is the declared interpreter, arrays plus wait -n/wait -p on supported versions make a bounded scheduler clearer.
Declare the interpreter and minimum version instead of writing Bash syntax under #!/bin/sh. Alternatively, use a purpose-built job runner that preserves argument boundaries and gives a manifest of job/result pairs.
Do not build commands by concatenating inputs. Pass each value as a quoted argument. If filenames are transported through a list, use a NUL-delimited protocol where the tools support it; newline-delimited lists cannot represent every pathname.
Decide fail-fast versus collect-all
A collect-all batch lets independent jobs finish and reports every failure. A fail-fast batch stops launching new work and signals active jobs after the first failure. Sending TERM only to recorded wrapper PIDs may leave grandchildren alive; process groups or a supervisor are needed for a real tree-wide cancellation policy.
Install traps before launches, keep the active PID set current, forward intended signals, wait for cleanup, and preserve the original failure status. Never use broad kill 0 without understanding that it targets the caller’s process group and may include the interactive shell or unrelated work.
Give each job a separate temporary directory and log. Concurrent append to one log can interleave lines or partial writes. The parent can print logs in input order after completion or attach job IDs/timestamps through a structured logger.
Verify scheduler semantics
Test zero jobs, one job, more jobs than the limit, fast failure, slow failure, a child spawning descendants, parent interruption, log-path errors, and inputs with spaces/newlines. Assert maximum observed concurrency and the final status policy. Use fake workers with controlled sleep and exit codes so the test is deterministic.
Shell parallelism is safe when every launched unit has an identity, resource bound, output boundary, recorded status, and termination path. & creates concurrency; the bookkeeping around it creates a trustworthy batch.
Related:
- Fixing a Terminal Left Without Echo or Line Editing After a Command Crashes
- Here-Documents in Shell: Expansion, Delimiters, Tabs, and Temporary Data
Sources: