PowerShell ForEach-Object -Parallel: Runspaces, Throttling, and Shared State

ForEach-Object -Parallel is convenient for independent work that spends substantial time waiting on I/O or can use multiple CPU cores. It is not a faster spelling of an ordinary foreach loop. Each parallel script block runs in a separate runspace on another thread, so startup, synchronization, object transport, and scheduling have real costs. Small operations often run slower when parallelized.

PowerShell 7.0 introduced this parameter set. PowerShell 7.0 created a new runspace for every iteration; starting in 7.1, the default is a runspace pool that reuses runspaces. The default throttle limit is five concurrent script blocks. These version details affect both performance and isolation assumptions, so test on the minimum supported engine rather than reasoning from a single current workstation.

Start with independent work and a sequential baseline

Parallel processing is safest when each item can be processed independently and the result is associated with an explicit input key. First implement and measure the sequential version. Only then compare it with a bounded parallel form using representative data and the same correctness assertions.

$targets = @('node-a.example', 'node-b.example', 'node-c.example')

$results = $targets | ForEach-Object -Parallel {
    $target = $_
    try {
        $reply = Test-Connection -TargetName $target -Count 1 -Quiet
        [pscustomobject]@{
            Target  = $target
            Success = [bool]$reply
            Error   = $null
        }
    }
    catch {
        [pscustomobject]@{
            Target  = $target
            Success = $false
            Error   = $_.Exception.Message
        }
    }
} -ThrottleLimit 3

$results | Sort-Object Target

This example uses an illustrative network probe; it is not a substitute for an application-specific health check. The important contract is one output record per input, with a stable identity and explicit result fields. Parallel output order is not a completion-order guarantee. If a consumer needs deterministic display or serialization, include the input identity and sort after all work is complete.

The sequential comparison should include elapsed wall time, memory use, remote service limits, and failure behavior. A higher -ThrottleLimit can overwhelm a remote endpoint, a local disk, or a process quota. CPU-count-based guesses are not enough for network-bound workloads, and an unbounded “all items at once” design can turn a transient slow service into a thundering herd.

Understand runspace reuse and scope

Use $using:Name to make a caller variable available inside the parallel script block. Values are references to objects, not automatically deep-cloned immutable snapshots. Reading an object that is not changing is usually straightforward. Mutating a shared object from several worker threads is not safe unless the object’s API supports concurrency or you add appropriate synchronization.

$prefix = 'probe'

$results = 1..8 | ForEach-Object -Parallel {
    [pscustomobject]@{
        Label = "$using:prefix-$_"
        Worker = [System.Threading.Thread]::CurrentThread.ManagedThreadId
    }
} -ThrottleLimit 4

The thread ID is diagnostic only; do not make correctness depend on a particular worker assignment. Pool reuse in PowerShell 7.1 and later means a runspace may process more than one iteration. Code should initialize per-item state inside the iteration instead of assuming every iteration starts in a pristine session. Use -UseNewRunspace only when the isolation requirement justifies its resource and setup cost; it is not a default performance optimization.

If workers must update shared state, choose a thread-safe collection such as ConcurrentDictionary or ConcurrentBag, or have each iteration return an immutable result and aggregate it after the pipeline finishes. A normal [System.Collections.Generic.List[object]] or hashtable is not made thread-safe merely by assigning it through $using:. Also avoid sharing mutable PowerShell objects whose methods are not designed for concurrent calls.

$observedGroups = [System.Collections.Concurrent.ConcurrentBag[string]]::new()

$items | ForEach-Object -Parallel {
    $bag = $using:observedGroups
    $bag.Add([string]$_.Group)
} -ThrottleLimit 4

$observedGroups | Group-Object -NoElement

The bag is a thread-safe collection, and aggregation happens afterward in the ordinary pipeline. Often an even clearer design is not to mutate shared state at all: emit one record per item, group those results serially, and let the output pipeline own aggregation.

Throttling, timeouts, and jobs

-ThrottleLimit caps simultaneous script blocks for that one invocation; it is not a process-wide or machine-wide concurrency budget. Multiple ForEach-Object -Parallel invocations can each consume their own limit. When -AsJob is used, each job has its own limit, so creating many such jobs multiplies the possible concurrency. Model the total fan-out across the whole script.

-TimeoutSeconds can stop running script blocks after a time budget and ignore remaining inputs. A timeout is a cancellation boundary, not a rollback transaction. Side effects already performed by a worker may remain. If a task changes external state, make it idempotent or design a compensating action; do not assume timeout restores the pre-run state.

$job = 1..20 | ForEach-Object -Parallel {
    Invoke-WorkItem -InputObject $_
} -ThrottleLimit 4 -AsJob

try {
    Wait-Job -Job $job -Timeout 120 | Out-Null
    Receive-Job -Job $job -Keep
}
finally {
    if ($job.State -in 'Running', 'NotStarted') {
        Stop-Job -Job $job
    }
    Remove-Job -Job $job -Force
}

Treat this as a lifecycle sketch, not universal production job management. Receive-Job behavior, output retention, child-job state, cancellation of external processes, and cleanup policy need to match the actual task. In particular, stopping a PowerShell worker does not guarantee that a separately launched external process has been terminated or that a remote operation was undone.

Errors and streams are nondeterministic

Workers execute concurrently, so the arrival order of success output, errors, warnings, verbose messages, and information records is nondeterministic. A terminating error normally ends that parallel instance, but other work items can continue. One worker’s exception is not automatically a cancellation request for every other worker.

Return structured per-item outcomes when partial success is acceptable. If the entire batch must fail atomically, parallel workers need a transaction or staging model that can validate every result before publishing changes. A simple try/catch around the pipeline cannot undo already committed external work. Avoid interpreting the order of diagnostic lines as the order in which operations began or completed.

Decide whether the work merits parallel execution

Use parallelism when there is enough work per item to amortize scheduling, runspace management, and object handoff. Network latency, file operations, and independent computations may benefit. Trivial property selection, string formatting, or a few arithmetic operations generally do not. Profile representative data, measure both the sequential and parallel paths, and retain parallel execution only if the end-to-end result improves without violating service limits.

A trustworthy test includes empty input, one item, more items than the throttle limit, a slow item, an expected failure, cancellation, and shared-state behavior. Verify that every input has one accounted-for outcome, that no hidden errors disappear in merged streams, and that the output remains correct after sorting. Test from a fresh pwsh -NoProfile process so profile aliases and functions do not change command resolution.

Make worker output observable without depending on timing

Workers are easier to support when their output is data rather than interleaved console text. Include a correlation key, outcome, elapsed time, and a sanitized error identifier in each returned record. Do not have every worker append to the same log file: concurrent appends can interleave, and a slow disk can become the bottleneck that removes the benefit of parallelism. If detailed logs are needed, buffer them per item and write them after the parallel stage, or use a logging sink documented to accept concurrent writes.

$items | ForEach-Object -Parallel {
    $item = $_
    $timer = [System.Diagnostics.Stopwatch]::StartNew()
    try {
        $value = Invoke-ReadOnlyCheck -Item $item
        [pscustomobject]@{
            ItemId = $item.Id
            State = 'Succeeded'
            ElapsedMilliseconds = $timer.ElapsedMilliseconds
            ErrorId = $null
            Value = $value
        }
    }
    catch {
        [pscustomobject]@{
            ItemId = $item.Id
            State = 'Failed'
            ElapsedMilliseconds = $timer.ElapsedMilliseconds
            ErrorId = $_.FullyQualifiedErrorId
            Value = $null
        }
    }
} -ThrottleLimit 4

The result object intentionally avoids serializing the full exception, which may contain paths, server names, request details, or secrets. Preserve enough diagnostic context in a controlled log, but apply the same redaction policy that you use for sequential scripts. If an item fails, decide explicitly whether the batch should continue, stop accepting new inputs, or cancel other work. Those policies are different and should be visible in code and tests.

PowerShell runspaces are separate execution contexts. A module, function, or variable loaded only into the caller’s runspace may not be available in a parallel worker the way a synchronous script author expects. Import required modules explicitly inside the worker or define a self-contained script block, and pass only the data each worker needs. Avoid depending on UI state, interactive prompts, caller-local aliases, or profile side effects. Such dependencies make a script difficult to run under a scheduler and can fail only after deployment.

Bound work admission as well as worker count. A ThrottleLimit of five caps active script blocks for that invocation, but the input producer, result objects, job output, and remote service may still accumulate state. For a huge input, process bounded batches or stream from a source that itself supports paging. If the downstream service publishes a request quota, size concurrency below that quota and include retries in the load estimate; retries can amplify an outage if every worker repeats immediately.

Cancellation should be cooperative where the operation supports it. Ctrl+C stops the ForEach-Object -Parallel command, and -TimeoutSeconds can stop the parallel run after a deadline, but already completed side effects remain. A script block that invokes a long-running native child or remote job may need its own cancellation and cleanup mechanism. Record which operations completed before cancellation, make retry safe through idempotency keys or deduplication, and avoid blindly re-running a partially completed batch.

Performance comparisons should measure end-to-end wall time, not only the loop body. Include data loading, runspace setup, network wait, serialization of results, sorting, and cleanup. Run several representative trials and compare median and tail latency. If the parallel implementation saves a few seconds but makes failure recovery, rate limiting, or output accounting much harder, keep the simpler sequential version unless the performance requirement justifies the extra operational complexity.

Related:

Sources: