Skip to content
macOSHow-To Published Updated 5 min readViews unavailable

Xcode Test Plans and `.xcresult`: Reproducible macOS Test Runs

Make Xcode test runs reproducible with schemes, test-plan configurations, stable .xcresult artifacts, and version-aware xcresulttool inspection.

A green test icon is a useful summary, but it is not a reproducible record of how a macOS test run was selected, configured, and executed. Xcode schemes define build and test actions; a test plan specifies which tests and configurations participate in the test action. The .xcresult bundle preserves the test session’s results and related data so a CI failure can be reviewed after the runner has been discarded.

The important unit of comparison is not only the test source. It is the tuple of Xcode version, scheme, plan, configuration, destination, resolved dependencies, and test selection. If one of these changes, two bundles may describe different experiments even when they have the same scheme name.

Make the test selection explicit

Start by identifying the active developer tools and the plans attached to the scheme:

xcodebuild -version
xcodebuild -scheme "DesktopApp" -showTestPlans

Apple’s current Xcode documentation describes test plans as documents associated with a scheme’s test action. A project can attach multiple plans to one scheme and mark one as the default; explicitly selecting a plan in CI avoids silently inheriting a different default after a project change. A plan can include or exclude targets, suites, individual test functions, and supported tagged tests. Xcode 16 and later also use Swift Testing tags for test-plan selection; tags supplement rather than replace source-level test suites.

Configurations let the same selected tests run under different settings. Depending on the Xcode release and test target, that can include launch arguments, environment variables, locale, simulated location, diagnostics, sanitizers, repetition behavior, coverage, or execution timeouts. More configurations improve coverage only when their distinct assumptions are intentional and recorded. They also increase runtime and can expose flaky tests whose result depends on order or shared state.

For an example macOS test action, create a new result bundle path for every invocation:

mkdir -p artifacts
RESULT="artifacts/DesktopApp-CI-$(uuidgen).xcresult"

xcodebuild test \
  -scheme "DesktopApp" \
  -testPlan "CI" \
  -destination 'platform=macOS' \
  -resultBundlePath "$RESULT"

Use the actual scheme and plan names reported by the project. In a workspace-based project, pass its workspace instead of assuming an .xcodeproj; choose a destination supported by the selected Xcode and scheme. -resultBundlePath must identify an output path that does not already contain a result bundle. A unique run-specific name avoids a stale bundle collision and preserves evidence from parallel jobs.

To narrow a diagnostic run, Xcode supports test identifiers with -only-testing; first discover the target and test names in the Test navigator or the generated test report. A narrowed run is useful for iteration, but it must not be reported as equivalent to the full test plan. CI should label the command and result artifact with the exact selection.

Treat .xcresult as an evidence bundle, not a stable directory API

When xcodebuild test runs from Terminal, it produces an Xcode result bundle with session results, logs, and code coverage when that feature was enabled. Open the bundle in Xcode’s Report navigator or use the installed xcresulttool to extract supported summaries and attachments. Apple introduced a newer command interface for result inspection in Xcode 16; Xcode 16.3 release notes document commands such as get test-results summary, get test-results tests, get test-results test-details, get content-availability, and export attachments.

Command names, options, output fields, and result contents are tool-version dependent. Before a script depends on one, record xcodebuild -version and inspect the installed command help:

xcrun xcresulttool help
xcrun xcresulttool help get test-results summary
xcrun xcresulttool get test-results summary --path "$RESULT"

The final command is an example for Xcode versions that expose the newer get test-results summary interface. If the installed version does not recognize it, use that version’s documented interface rather than parsing private bundle files or copying a legacy xcresulttool recipe. Apple’s release notes mark older result-inspection commands as deprecated in favor of the newer interface. Treat a JSON export as a report format whose shape can evolve, not as a permanent schema without versioning and tests.

For durable CI evidence, retain the raw .xcresult bundle, the exact xcodebuild command, Xcode build number, destination, test plan, dependency lock state, and the text build log. Hash the artifact after the run if auditability matters. Keep extracted summaries alongside it, but never replace the original with a redacted or transformed copy. Result bundles can contain logs, screenshots, attachments, paths, test data, and coverage; apply the same access control and retention policy used for other build artifacts.

Review results without hiding exclusions or environmental failures

The report can distinguish passed, failed, skipped, and expected-failure outcomes, but a top-level status should be read with the selected plan and configuration. A plan that excluded an integration suite did not verify that suite. A test action can also fail during build, simulator or test-runner setup, app installation, signing, or post-test processing before a test body executes. Inspect the test report’s Build, Tests, and Log sections before diagnosing the failure as an assertion regression.

When comparing two CI runs, compare only like-for-like inputs. A different Xcode build, runtime, architecture, localization, code-signing identity, environment variable, test ordering, or repetition mode can change behavior. Store those dimensions in the CI job summary. If parallel execution is enabled, make test fixtures and temporary storage unique per worker; do not use a shared mutable profile merely because the tests passed serially on a developer Mac.

Use diagnostics deliberately. Address Sanitizer, Thread Sanitizer, Main Thread Checker, and test repetition can find classes of issues that ordinary runs miss, but they alter timing and resource use. A sanitizer run is a distinct configuration, not a replacement for the release-like test action. Performance measurements need a consistent destination and a controlled baseline; a CI screenshot or a single best duration is not a performance trend.

The practical goal is a result another engineer can interpret without access to the original runner. Pin the toolchain, choose the scheme and test plan explicitly, preserve the raw result bundle and invocation metadata, and use only the xcresulttool interface supported by that Xcode installation. This turns “it passed once” into a traceable test event without pretending that a bundle alone makes an unrepeatable environment deterministic.

Related:

Sources:

Comments