Skip to content
SRE & DevOpsDeep Dive Published Updated 6 min readViews unavailable

Google Cloud Build Step Graphs: Parallelism, Dependencies, and Failure Policy

Model Cloud Build steps as an explicit dependency graph, parallelize independent checks, and keep optional failures visible without weakening release gates.

Cloud Build configurations are often written as a list of commands, but the list is also an execution graph. A step’s id is its unique identifier, and waitFor declares which earlier steps must finish successfully before it can start. That distinction matters operationally: a build can be needlessly serial, accidentally concurrent, or report success even though a non-blocking check failed. Treating the graph and its failure policy as reviewable release logic makes these outcomes deliberate.

The default is a chain, not automatic parallel work

Without an explicit waitFor, a step waits for all earlier steps in the configuration to complete successfully. As a result, a simple list is sequential even when two checks do not depend on one another. To start a step when the build starts, set waitFor: ['-']; to join independent work later, name every required predecessor. A referenced step must have an id, and that ID must appear earlier in the steps list.

The graph below prepares a workspace, runs unit and integration checks concurrently, and creates a release artifact only after both checks succeed. An optional report starts independently and is explicitly non-blocking:

steps:
  - name: ubuntu
    id: prepare
    entrypoint: bash
    args: ['-e', '-o', 'pipefail', '-c', './ci/prepare.sh']

  - name: ubuntu
    id: unit-tests
    waitFor: ['prepare']
    entrypoint: bash
    args: ['-e', '-o', 'pipefail', '-c', './ci/test-unit.sh']

  - name: ubuntu
    id: integration-tests
    waitFor: ['prepare']
    entrypoint: bash
    args: ['-e', '-o', 'pipefail', '-c', './ci/test-integration.sh']

  - name: ubuntu
    id: package
    waitFor: ['unit-tests', 'integration-tests']
    entrypoint: bash
    args: ['-e', '-o', 'pipefail', '-c', './ci/package-release.sh']

  - name: ubuntu
    id: license-summary
    waitFor: ['-']
    entrypoint: bash
    args: ['-e', '-o', 'pipefail', '-c', './ci/optional-license-summary.sh']
    allowFailure: true

The ubuntu image name follows Google’s build-configuration examples; for an actual release pipeline, replace floating builder references with a reviewed image reference, preferably an immutable digest, and verify that it contains the expected shell and tools. Also make sure the scripts are present in the build workspace and that parallel checks do not race while mutating the same generated files. Cloud Build automatically makes /workspace available across steps; parallel readers are usually straightforward, while concurrent writers need isolated output paths or explicit serialization.

Design dependencies from release invariants

Begin with the artifact that is allowed to move toward deployment. Work backward: which checks must be green before it is assembled, and which inputs do those checks require? Then place independent read-only checks beside one another. Do not add a dependency merely to preserve visual list order; every edge can increase critical-path time. Conversely, do not omit an edge because two commands happen to pass in a small test run.

The package step above names both test IDs, so it cannot begin until both are successful. A failure in either required test keeps that artifact step from becoming a release candidate. If a check generates a file consumed by another step, declare the producer as a dependency even if the file is in the shared workspace. This makes the required ordering explicit instead of relying on a scheduling coincidence.

Keep the graph acyclic and reviewable. IDs should describe stable roles such as unit-tests or publish-image, not transient shell details. Each waitFor reference should be easy to trace to one earlier ID. A reviewer should be able to answer, from the YAML alone, whether artifact creation is gated by every required validation and whether a deployment can start before provenance, policy, or test results exist.

A successful build can still contain a failed step

allowFailure: true says that a step’s failure does not fail the overall build as long as the other steps succeed. Google Cloud Build still records that permitted step as FAILED; a green build status is not the same thing as every step being green. allowExitCodes is narrower: only listed exit codes are tolerated, and it takes precedence over allowFailure when both are set. The schema documents an upper bound of 255; list only explicit nonzero codes that the pipeline intentionally treats as acceptable.

Use these controls only for work that is genuinely optional to the release decision, such as an informational report whose absence cannot make the artifact unsafe or invalid. Do not apply them to unit tests, artifact verification, required security policy, signing, or deployment authorization. Avoid putting optional jobs in the dependency path of a required job unless you have verified and documented the interaction you want. The example’s report has no downstream dependency, so its failure remains visible without serving as a release gate.

For an exit-code exception, document the exact condition that produces the tolerated code and make the script emit a separate machine-readable result. For example, an unavailable optional data source should not be encoded with the same exit status as a parser defect. A broad allowFailure can turn both into an overall successful build; a narrow exception plus a visible step result gives operators better evidence.

Observe both the graph result and the overall status

After a build, inspect the overall status and per-step statuses together. This is especially important when a configuration permits failure: Cloud Build may report SUCCESS while a permitted step remains FAILED. The build record is therefore the evidence to retain, not only a badge or the final command-line exit code.

gcloud builds describe "$BUILD_ID" \
  --project="$PROJECT_ID" \
  --region="$REGION" \
  --format=json

For each required check, verify that its ID is present and that its status is SUCCESS; inspect timing and logs when a step is slow or unsuccessful. For optional work, record whether it was skipped, failed with an allowed result, or succeeded. Attach the build ID, resolved source revision, configuration revision, and artifact digest to the release record so a later operator can reconstruct what the graph actually ran.

Before adopting a new graph, test at least four cases in a non-production trigger: all required steps pass; one required check fails; the optional step fails; and one required check is deliberately delayed. Confirm that a required failure prevents packaging, that packaging waits for a delayed required check and proceeds only after it succeeds, and that an optional failure does not block packaging. Also verify that the build’s final status matches the policy and that the failed optional step remains visible. This exercises both the dependency graph and the result interpretation, not just whether the YAML parses.

Review checklist

  • Every waitFor ID exists earlier in the configuration, and each step’s dependencies match its real inputs.
  • Independent steps are read-only or write to separate paths; shared workspace state is not mistaken for an ordering guarantee.
  • Release-critical tests and policy checks have no tolerated exit path.
  • Each tolerated failure has an owner, reason, and observable per-step result.
  • The artifact or deployment step waits for every required validation and uses the exact output being validated.
  • The build record, logs, source revision, and artifact digest are retained for later review.

Parallelism is a performance feature, not a substitute for dependencies. A well-designed Cloud Build graph shortens only the work that can safely overlap and keeps the path to a publishable artifact strictly gated.

Related:

Sources:

Comments