Apache Beam in WSL: DirectRunner Semantics and Local Pipeline Tests

Apache Beam separates pipeline logic from the runner that executes it. The DirectRunner runs a pipeline within the local process and is designed to validate that code follows Beam’s model, including checks such as element immutability, encodability, and serialization of user functions. That makes it valuable for unit and small integration tests in WSL. It is not a production runner and does not demonstrate distributed throughput, remote worker dependencies, or behavior specific to Flink, Spark, or a cloud service.

Use a Linux Python environment inside WSL and keep the repository, virtual environment, fixtures, and generated files in the distro filesystem. Microsoft recommends the Linux filesystem for Linux tools because the Windows mount boundary can change I/O and file behavior. If a test reads a Windows file, make that an explicit fixture boundary and keep the same test separate from a baseline DirectRunner test.

Install the SDK and make runner choice explicit

Follow the Apache Beam installation guide for the exact SDK, Python release, and platform. Beam SDK and runner artifacts have their own compatibility constraints. Record the Beam version, Python version, and runner selection with each test. Do not assume that an environment with Apache Beam installed can launch a remote runner without its separate cluster credentials, packages, and service configuration.

For a small model, make the runner explicit in options instead of relying on a default that may differ between environments. A small Python example creates two values, maps them, and uses a test assertion:

import apache_beam as beam
from apache_beam.testing.util import assert_that, equal_to

with beam.Pipeline() as pipeline:
    values = pipeline | "Create" >> beam.Create(["a", "bb"])
    lengths = values | "Length" >> beam.Map(len)
    assert_that(lengths, equal_to([1, 2]))

The Python SDK’s local pipeline can use DirectRunner for this kind of small test. The assertion validates the output as a collection, not the order in which distributed elements would arrive. If order is part of the application contract, encode it in keys or timestamps and use a transform with documented ordering semantics; do not rely on the order shown by one local run.

Test Beam model assumptions, not only a happy path

The DirectRunner deliberately performs correctness checks that help catch assumptions invalid under the Beam model. A local pipeline can reveal that a user function mutates its input, cannot be serialized, or depends on arbitrary element order. Write unit tests around pure transforms and use Beam’s testing utilities to assert expected outputs. Keep test collections small enough for local memory; the runner is optimized for correctness and validation, not performance.

The same test should include edge cases: empty input, duplicate keys, null or missing fields, malformed records, late timestamps if the pipeline uses event time, and the expected behavior after a failed parse. Do not use a local generated dataset so small that every record fits in a single bundle and then infer that your aggregation or stateful transform works at scale. A representative test verifies semantics; scale and runner-specific behavior need a separate environment.

For windowing and triggers, test event timestamps explicitly instead of relying on processing time. A local bounded collection can validate basic window assignment, but it does not reproduce the timing, watermark progression, source behavior, and checkpoint lifecycle of a live distributed runner. Use Beam’s test stream facilities when simulating unbounded inputs, watermarks, and processing-time advances, and consult the SDK documentation for the available controls in the selected Beam version.

Keep pipeline configuration portable

Separate pipeline transforms from runner options, storage paths, and credentials. A pipeline can be portable in the Beam model while still relying on runner-specific capabilities, container images, side inputs, or I/O connectors. Keep options in a small explicit configuration object and provide defaults suitable for local tests. Do not embed cloud credentials or host-local absolute paths inside a DoFn.

Run local tests from the Linux interpreter used by WSL CI or development. Print the Python executable and import Beam before debugging pipeline code. If a test passes in one shell and fails in another, compare package versions, entrypoint, current directory, and environment variables. File watchers and symlinks crossing /mnt/c can also change how fixtures appear to Linux code.

When the pipeline uses an I/O connector, validate the connector’s local setup separately. A DirectRunner test that uses an in-memory Create transform does not prove authentication, schema discovery, network reachability, or remote write semantics. Use a disposable source and destination for integration tests, and do not reuse a production topic or table just because the runner is local.

DirectRunner tests should avoid depending on incidental execution order. Beam may process elements in an order that is not the order they entered the pipeline, and distributed runners can make that assumption even less safe. For assertions over unordered output, compare normalized collections or use Beam’s testing helpers with an explicit matcher. If order is a business requirement, model it in the data and transform graph instead of expecting the runner to preserve input order. Also test empty input and duplicate keys; a pipeline that only passes a small, non-empty fixture can miss initialization and grouping edge cases.

Move from local tests to a runner-specific verification

The DirectRunner is a stage in a validation ladder. Start with unit tests for individual functions, run a small pipeline under DirectRunner, and then execute the same transform graph on the target runner with a bounded input. Compare outputs, error behavior, and metrics. Runners can differ in supported capabilities and operational settings; consult the Beam capability matrix and runner-specific docs rather than assuming every API has identical support.

Do not benchmark a local pipeline as a proxy for a distributed runner. The DirectRunner executes inside the process that constructed the pipeline and must fit user data in local memory. Its extra correctness checks and local scheduling do not model remote serialization costs, network shuffle, autoscaling, worker loss, or external checkpoint storage. Measure performance on a representative deployment using safe test data.

WSL troubleshooting and cleanup

If worker functions fail to serialize, move closures and local state out of nested functions and confirm dependencies are importable in a clean process. If an assertion fails intermittently, examine nondeterministic order and avoid treating a set of values as an ordered list. If memory grows, reduce test data or run smaller windows; do not raise the VM-wide memory cap to hide an unbounded unit fixture.

If a local I/O test cannot read a file, print the resolved path and check Linux permissions, mount type, and case sensitivity. When a pipeline produces local outputs, write them under a dedicated test directory and remove only that directory after the process exits. Keep logs and test seeds so an intermittent failure can be reproduced.

For keyed aggregations, assert the mathematical result without assuming collection order. Convert output to a canonical mapping or sort by a stable key in the assertion. Exercise a key with multiple values and one with a single value so combine logic is tested across both cases. If the transform uses a custom coder, validate encoding and decoding directly and include a round-trip test. These checks catch data-model problems before runner setup obscures them.

Package dependencies deserve their own test. A local DoFn can import a module from the developer’s shell that is absent from the submitted worker image or remote environment. Declare third-party dependencies using the runner’s documented packaging mechanism and run a clean-environment test that does not inherit the developer’s site packages. A local pass with the full workstation environment is useful feedback, but weak evidence that the serialized pipeline can execute elsewhere.

Acceptance criteria

Accept the WSL Beam environment when the SDK and Python versions are recorded, runner selection is explicit, a deterministic pipeline and output assertion pass on small fixtures, and tests exercise empty input and at least one relevant edge case. Confirm that no test depends on input ordering unless the pipeline defines it, and list runner-specific behaviors still requiring verification.

DirectRunner is a local correctness and development tool. It is not a production runtime, a throughput benchmark, or proof that a distributed runner has the same capabilities and failure behavior.

Related:

Sources: