systemd-sysext and confext: Reversible Runtime Extensions for Immutable Linux

An immutable or image-based Linux host sometimes needs a temporary tool, a narrowly scoped configuration overlay, or a site-specific file without rebuilding its base image. systemd’s system extensions (systemd-sysext) and configuration extensions (systemd-confext) provide a runtime merge mechanism for those cases. They are not package managers, containers, or a security boundary. The distinction matters: a successful merge can make extension files appear under the host’s normal paths while leaving the original base image untouched, but those files then execute with the host’s ordinary security context.

This guide focuses on deployment and rollback semantics. Use the manual installed on the target system as the authority for supported commands and version gates. The online latest manual describes current upstream behavior, while distribution builds may ship older systemd versions or downstream changes.

Choose the extension mechanism by filesystem boundary

systemd-sysext combines an extension’s /usr/ and /opt/ trees with those same host hierarchies. The extension is discovered as a directory or supported disk image, then merged through OverlayFS. Unmerging removes that overlay mount and reveals the unmodified host tree again. Files in other paths inside a sysext image, including /etc/ and /var/, do not become part of those host hierarchies.

systemd-confext applies the same general idea to /etc/ only. It is for configuration files, not binaries under /usr/. Its merged /etc/ is mounted with nosuid and, by default, noexec; check the local manual if an application depends on different mount flags. Keep configuration ownership and precedence explicit rather than treating a confext as another writable copy of the host’s /etc.

Both mechanisms may use a directory tree, a Btrfs subvolume, or an appropriately formatted raw disk image. Sysext images are searched in /etc/extensions/, /run/extensions/, and /var/lib/extensions/; confext images use their own documented directories. The system extension service can activate installed images during boot after their underlying filesystems are mounted. Do not put content required by the earliest boot stages in an extension that is only available later.

Compatibility metadata is part of the image contract

A sysext image normally includes /usr/lib/extension-release.d/extension-release.NAME, with NAME matching the image name; a documented user.extension-release.strict=0 extended attribute can relax that filename check when exactly one matching release file is present. A confext uses the corresponding file under /etc/extension-release.d/. The release file is not decoration: systemd compares it with the host’s os-release metadata before merging.

For an extension whose ID= is not _any, that value must correspond to the host. Its defined SYSEXT_LEVEL= must match; when it is absent, VERSION_ID= is compared instead. An optional ARCHITECTURE= must use a systemd architecture identifier and match the running kernel architecture unless it is _any. Confext images use CONFEXT_LEVEL= rather than SYSEXT_LEVEL=. A broadly compatible image should not claim compatibility merely because its directory tree happens to mount.

For example, this minimal metadata is valid only after replacing the sample values with values that actually match the target host:

ID=example
VERSION_ID=1

If the host and extension use SYSEXT_LEVEL=, set and compare that field according to the local os-release and extension-release contract instead of assuming that VERSION_ID= is always the compatibility key. Test the extension against each supported base image and architecture before promotion.

Build additively and keep the base image authoritative

Extensions are intended to add files that are absent from the base OS. OverlayFS can technically hide or replace files in lower layers, but upstream systemd recommends against using system extensions as an override or removal mechanism. Replacing a host binary this way can create an untracked version split: the package database still describes the base package while the merged tree executes a different file.

There is no general dependency resolver for sysext images. Build an extension with the files it requires, except for libraries and resources intentionally supplied by the exact base OS image. Produce the extension and base image through a compatible build pipeline, record their digests, and retain the extension as a separate artifact. A directory under /var/lib/extensions/ is convenient for a test, but the final artifact should still have a reproducible manifest and provenance.

Inspect the base and extension metadata before any merge. A safe release procedure uses a disposable VM or an isolated root image, not a developer workstation or production host:

systemd --version
cat /usr/lib/os-release
systemd-sysext --help
systemd-sysext list
systemd-sysext status

Confirm the exact systemd version and release identifiers, then stage one extension and check that its extension-release filename, ID=, version-level fields, and architecture match that test system. Review the merged paths before enabling the same artifact in a controlled environment. A successful compatibility check is not a security review of the files in the extension.

Understand merge, refresh, and unmerge as mount operations

systemd-sysext merge combines the discovered extension trees with /usr/ and /opt/. status reports whether the hierarchies are currently merged, and list reports discovered images. unmerge removes the active sysext overlay. refresh rebuilds an already merged view after extension images change.

Treat these as privileged host filesystem operations. refresh temporarily unmounts the old overlay before installing the replacement, so extension-provided resources can disappear briefly. Processes that start or open files during that interval may fail. Coordinate refresh with the services that consume extension files, and test the exact upgrade sequence. Do not copy a running system extension over its current directory and assume the mounted view has changed; after changing images, use the supported refresh path and verify the result.

If automatic boot activation is enabled, installed images are considered together at boot. There is no separate enable/disable state for each image. The manual documents masking an image by placing an empty directory with the same name in /etc/extensions/. Prefer an explicit manifest and a reviewed mask over deleting or renaming extension artifacts ad hoc, especially when more than one search directory is in use.

For an isolated root tree, --root= makes the command operate below that root rather than on the live host hierarchy. This is useful for controlled image testing, but it is not a magic sandbox: the root tree must be disposable, and the command still needs the kernel and mount permissions required by the operation. Never test merge, refresh, or unmerge against the wrong root path.

Mutability changes the rollback story

By default, sysext and confext images are read-only. On a mutable host, the merged /usr/, /opt/, or /etc/ hierarchies are also made read-only unless mutability is explicitly enabled. Recent systemd versions support several --mutable= modes, including automatic routing, imported write-routing directories, and ephemeral modifications. These modes are version-gated; use the installed manual and inspect the exact mode before relying on it.

Mutable writes may be routed below /var/lib/extensions.mutable/. That means a file written through the merged view may outlive an unmerge, depending on the selected mode and write-routing configuration. Ephemeral modes are intended to discard their routed changes after unmerge, but they are still not a substitute for a documented data-retention policy. Keep persistent state in an explicitly managed location, not in an extension’s executable tree, and test rollback by inspecting both the merged hierarchy and the backing write-routing directories.

Disk-image extensions can optionally carry Verity authentication information. Do not infer integrity or authenticity from the .raw suffix, a matching extension name, or a successful merge. Inspect the effective image policy, verify artifact provenance and digest, and decide explicitly which signing or Verity guarantees your boot and deployment chain requires. Compatibility metadata answers whether an image matches a host; it does not establish who built the image or whether it is trustworthy.

Do not ship services or security isolation through sysext

System extensions become part of the host’s /usr/ and /opt/ view; they do not isolate the code they provide. For resources needed in the earliest initrd stages, systemd provides separate systemd-sysext-sysroot.service and systemd-confext-sysroot.service units; their documented limitations include split-/var layouts, so verify the target boot design instead of assuming the regular post-mount activation service is early enough. For packaged daemons that need a separately managed unit and service-level sandbox profile, portable services are another option: portablectl attaches service units from an image and applies a profile. Portable services are not full containers either, but have a distinct service lifecycle and access restrictions.

Choose based on the contract:

Use sysext for additive OS resources that should appear in the host filesystem, such as optional diagnostics on an immutable machine.
Use confext for controlled configuration under /etc/, with explicit compatibility and mount-flag review.
Use a portable service when the deliverable is a daemon that should run as a managed service with a separately reviewed sandbox profile.
Use a container or VM when the workload requires a stronger operating-system isolation boundary.

Do not describe a sysext image as a container or claim that extension-release compatibility fields make it safe. The mechanism composes filesystems; your package verification, update control, and runtime policy remain separate responsibilities.

A production readiness checklist

Before promotion, record the host image digest, systemd version, extension digest, release metadata, architecture, expected merged paths, and the exact commands used to merge and roll back. Verify that every extension file is additive and that no critical boot dependency is introduced after its required startup phase. Exercise a failed version match, a missing image, a malformed release file, an extension masking case, and refresh during a controlled maintenance window. Observe services and logs before and after unmerge; disappearance of the extension’s files is expected, while unexplained changes to base files are not.

Keep the extension image immutable in normal operation. Sign or authenticate it according to the host’s image policy, promote the exact tested digest, and retain a reversible deployment record. If the deployment cannot prove which base image and extension produced the merged filesystem, it is not reproducible enough for production.

Related:

Sources: