BuildKit Cache Strategy for CI: Layers, Mounts, and Trust Boundaries

BuildKit can reuse prior build results at several layers: completed Dockerfile instructions, temporary package-manager cache mounts, and exported caches shared between otherwise ephemeral CI builders. These mechanisms solve different problems. A layer cache avoids executing an unchanged instruction; a cache mount preserves tool downloads even when the instruction must run again; a remote cache transports reusable results between builder instances.

None of these caches proves that an image is current, reproducible, or trustworthy. In particular, a cached RUN apt-get update does not re-query package repositories just because time has passed, and changing a mounted secret does not invalidate the instruction cache. Treat caching as a performance optimization over a separately defined dependency-update and release process.

Understand what invalidates an instruction

BuildKit evaluates the Dockerfile in order. For most instructions, it reuses a result when the instruction and its relevant inputs match a previously cached build. A change invalidates that step and later dependent steps. COPY, ADD, and build-time bind mounts also depend on file metadata for the files involved; a change to file modification time alone does not invalidate their cache checksum.

The builder does not inspect arbitrary files changed inside an earlier RUN layer to decide whether a later RUN is current. For example, if RUN apt-get update has the same command and cache inputs as before, the cached result can be reused even when the upstream repository now contains newer package metadata. Rebuilding the same Dockerfile next week is therefore not a dependency refresh policy. Use lockfiles and explicit version constraints where available, and schedule or trigger rebuilds that refresh the base image and dependency data. --pull refreshes the base-image reference; it is not equivalent to invalidating all build steps. Use --no-cache or --no-cache-filter <stage> when the goal is to execute selected build steps again, understanding the additional cost.

Order instructions around how frequently their inputs change. Copy dependency manifests and lockfiles before application source so an edit to a source file does not unnecessarily invalidate dependency installation. Keep the build context small with .dockerignore; files sent as context can participate in cache inputs and may be copied into intermediate layers. Exclude local build output, repository metadata, credentials, and files the build does not need.

# syntax=docker/dockerfile:1
FROM python:3.13-slim AS build
WORKDIR /src

# Dependency installation is reused until its manifests change.
COPY requirements.txt ./
RUN --mount=type=cache,target=/root/.cache/pip \
    python -m pip install --prefix=/install -r requirements.txt

# Frequent source edits invalidate the application build, not the dependency layer.
COPY . ./
RUN python -m compileall -q .

The cache mount keeps pip’s downloaded packages available to later builds. The pip install result is still written under /install and captured by the instruction’s build result; the mount is not a substitute for installing dependencies into a path that will be copied into the final image. Review the chosen base-image tag and dependency constraints separately; this example demonstrates cache structure, not a complete production Dockerfile.

Use cache mounts for package-manager state

A RUN --mount=type=cache mount provides persistent mutable storage to a build instruction without placing the cache directory itself in the resulting image layer. It can reduce network downloads when the instruction has to run again, for example after an application dependency lockfile changes. The application or package manager must still write its actual installation output outside the cache mount if that output belongs in the image.

Choose the mount target based on the package manager’s documented cache location. For tools that cannot safely use one cache directory concurrently, configure an appropriate sharing mode such as sharing=locked; Docker’s package-manager examples use locking for APT cache directories. Cache mounts are mutable performance state, not an integrity boundary. A build should remain correct if the mount is empty, evicted, or populated by a prior build; verify packages using lockfiles, checksums, signatures, or the package manager’s normal validation rather than trusting the cache contents alone.

Build-time bind mounts have a related but distinct purpose: they expose context files to a single RUN without adding those files as a persistent image layer. Bind mounts are read-only by default. Writes made to them are discarded when the instruction ends, so write generated artifacts outside the mount and deliberately copy the outputs that belong in the image. Use bind mounts when source is needed only to produce an artifact; use COPY when files must persist into a later layer or final image.

Export remote cache deliberately in CI

Each BuildKit builder has its own local cache. Ephemeral CI workers commonly start with an empty builder, so import a remote cache and export the new result explicitly with --cache-from and --cache-to. A registry cache is a common portable option:

docker buildx build --push \
  --tag registry.example.com/platform/service:commit-abc123 \
  --cache-from type=registry,ref=registry.example.com/platform/service:cache-main \
  --cache-to type=registry,ref=registry.example.com/platform/service:cache-feature,mode=max \
  .

Treat the example names as a cache design, not literal tags to reuse. Cache exporters write to a location and can overwrite existing cache data at that location. Use separate, intentional destinations when maintaining branch-specific caches; import both the branch cache and a trusted baseline cache when that improves hit rate. Define who can write each cache reference and whether untrusted pull-request jobs can read from or publish to it. Do not let an untrusted build overwrite the release branch’s trusted cache namespace.

The registry backend can store cache in a separate image reference from the application image. Buildx also supports inline, local, and other remote backends, each with different driver and storage requirements. Check the current Buildx documentation for the actual driver and image-store capabilities on the CI worker instead of assuming that every backend is available with every builder configuration.

For exported caches that support mode, min includes layers exported into the resulting image, while max also includes intermediate build stages. max can improve cache reuse for multi-stage builds, but it may make the cache larger and expose more intermediate build data to every principal with cache-read access. Choose the mode after measuring import/export time and storage use, and protect the cache reference at least as carefully as other private build artifacts.

Treat GitHub Actions cache as an explicit CI backend

The Docker gha cache backend uses GitHub Actions cache storage and is useful for workflows running in GitHub Actions, subject to its current availability, size, usage, and rate limits. Docker’s current documentation labels this backend experimental and notes that the default Docker driver requires the containerd image store for it. Check those conditions for the version and runner image in use before making it a release dependency.

The backend’s scope identifies a cache object. Its default is buildkit; if multiple image builds write the same default scope, later exports can replace earlier cache state. Assign a distinct scope to each image or build target, then import the appropriate current-branch and trusted base-branch scopes. GitHub’s cache access rules still apply: a workflow does not have unrestricted access to every branch’s cache, and cache entries may be evicted. A cold-cache build must remain correct and acceptable, even if it is slower.

Observe cache import and export duration, cache hit rate, and build duration separately. A build can be correct but slow because the cache is unavailable; a build can be fast and still contain stale dependencies because it reused a valid-but-old RUN result. If the cache API is throttled, storage is evicted, or export timeouts recur, reduce duplicate cache lookups, narrow scopes, or use a different supported backend rather than treating cache export failure as evidence that the image itself is invalid.

Keep credentials out of image layers and cache metadata

Do not pass build credentials through ARG, ENV, or COPY. Docker warns that build arguments and environment variables can persist in the final image. Use BuildKit secret or SSH mounts so a credential is made available only to the instruction that needs it:

RUN --mount=type=secret,id=package_token,env=PACKAGE_TOKEN \
    package-client fetch --locked

Pass the secret through the build client’s supported --secret option and ensure the command does not echo it into logs or write it into a generated file that is later committed to an image layer. Secret mounts reduce accidental persistence; they cannot prevent a build command from copying a secret into output, and they do not make a malicious or untrusted Dockerfile safe to execute with powerful credentials.

Secret contents are not part of the cache checksum. Rotating a token therefore does not automatically cause the associated RUN instruction to run again. If a changed credential must force a re-fetch, use an explicit non-secret version value or a targeted cache invalidation mechanism. Never use the credential itself as a cache-busting build argument. Conversely, do not assume that a cache hit proves the secret was valid during the current run: the command may not have run at all.

Separate cache correctness from release verification

Test the same Dockerfile with an empty cache and a warm cache. The resulting image should meet the same functional tests in both cases. Test a source-only change, a lockfile change, a base-image update, and a secret rotation independently so you know which steps rerun and which outputs remain cached. Inspect build logs for the expected cache-hit sequence; do not infer it from total build time alone.

For a release, independently verify the resulting image digest, provenance, and software inventory under the release policy. Cache reuse does not establish that a base image received current patches, that dependencies match policy, or that the artifact came from a trusted builder. If an integrity incident is suspected, rebuild from a clean builder with the required dependency-refresh settings, preserve the prior cache and build records as evidence, and compare the produced digest and attestations rather than silently pruning the only diagnostic state.

Related:

Sources: