Skip to content
Shell & TerminalDeep Dive Published Updated 5 min readViews unavailable

Nushell par-each: Output Ordering, Thread Budgets, and Side Effects

Use Nushell par-each for independent pipeline work while controlling worker threads, preserving row order, and preventing side effects from racing.

Nushell’s par-each is a structured parallel counterpart to each: it applies a closure to rows in a list or table and collects the returned values. That makes it useful for independent, relatively slow work such as reading several directories or running bounded per-item checks. It is not a promise that every pipeline becomes faster, that side effects happen in input order, or that a worker may safely mutate shared state.

The most important operational difference is ordering. Unless --keep-order is selected, results can arrive in completion order rather than input order. The command also provides --threads to set the worker count. Choosing those policies explicitly prevents a later change in input size or hardware from changing how a report is interpreted.

Use parallel mapping only for independent rows

each is the serial map operation. par-each runs multiple closure applications at once using Nushell’s thread pool. For short arithmetic or string transformations, scheduling overhead can outweigh the work; benchmark representative input before replacing a clear serial pipeline.

This example counts directory entries from a list of directory rows. Each closure produces one record and only reads the directory named by its row:

let summaries = (
    ls
    | where type == dir
    | par-each --threads 4 --keep-order {|row|
        {
            path: $row.name,
            entries: (ls $row.name | length)
        }
    }
)

$summaries | sort-by path

The example bounds parallel workers and preserves input order before the explicit final sort. It does not create an atomic filesystem snapshot: files may be added or removed while the closures run. If a report needs a stable point-in-time view, coordinate with the producer or operate on a filesystem snapshot rather than treating a directory listing as transactional.

Use the closure’s row value as its input and return a record that retains the row’s identity. A bare result such as a count loses the path that produced it. Keeping an ID beside each result makes unordered completion safe to reconcile when input order is not needed.

Make ordering a deliberate part of the interface

Without --keep-order, par-each does not promise that result position corresponds to input position. Completion order can vary with filesystem cache state, external command latency, thread scheduling, and system load. If position itself carries meaning, use --keep-order. If the data already has a stable key, preserve that key in every result and sort or join by it after parallel work.

let checks = (
    [alpha beta gamma]
    | enumerate
    | par-each --threads 3 {|item|
        {
            input_index: $item.index,
            name: $item.item,
            normalized: ($item.item | str upcase)
        }
    }
    | sort-by input_index
)

$checks

This pattern does not rely on worker completion order. The input index is explicit data, and the final sort reconstructs presentation order. For workloads where order is a contract, using --keep-order is simpler; for keyed records where the output order is a presentation choice, carrying an identity can make the contract clearer.

Bound concurrency and external work

--threads is an operational limit for a par-each invocation, not a global budget across the whole machine. If multiple Nushell commands or other applications run concurrently, their combined resource demand can exceed the setting. Nested or overlapping parallel calls also deserve measurement; do not assume a thread count automatically provides a process-wide cap for child processes or network requests.

When a closure launches an external command, account for both Nushell workers and the command’s own threads. Four parallel closures that each start a multithreaded compiler, image converter, or compressor can create far more runnable workers than four. Choose a small explicit bound, measure CPU and memory, and include a cancellation or deadline policy for long-running children.

Keep each worker’s side effects isolated. Writing all rows to one shared output file can interleave or overwrite data; instead, return structured values to one downstream writer, or assign unique per-row temporary paths and merge them in a separate serial step. Do not use shared mutable state as an ordering mechanism. If an external service has rate limits, add an explicit rate-limited work queue rather than assuming a thread count is equivalent to requests per second.

Treat errors and partial reports explicitly

A report is only complete when the caller knows which inputs were attempted and which produced results. Retain a key for each row, test missing paths and permission failures, and decide whether a failed row aborts the operation or appears as a typed error record. Avoid converting every failure to null if downstream code might mistake a missing result for successful absence.

Exercise the pipeline with empty input, one row, more rows than worker threads, a deliberately slow row, a missing directory, and a permission-denied path. Repeat the slow-row case several times and verify that unordered mode is not accidentally consumed as ordered data. For an external-command closure, capture the command’s exit status and stderr using Nushell’s documented external-command behavior, and keep that failure separate from the record’s successful rows.

Nushell’s documentation describes a dedicated thread pool for par-each, reusing idle top-level pools and creating separate pools for nested or overlapping calls. That is an implementation detail worth considering when measuring nested work, but it is not a substitute for the command’s documented flags or an application-level resource budget. Recheck behavior against the Nushell version deployed in production.

par-each is best viewed as parallel mapping over independent values. Preserve identity, state the ordering contract, constrain the work per invocation, and collect failures in a form the next pipeline stage can audit. Those four decisions make parallel execution testable rather than merely faster on one machine.

Related:

Sources:

Comments