Fixing FreeBSD ZFS Boot Environments That Consume Unexpected Pool Space

A ZFS boot environment is usually a clone of a root snapshot. At creation it shares blocks and appears cheap; as the active and preserved environments diverge, snapshots keep old blocks referenced and unique usage grows. Deleting files from the active root can therefore free little pool space because another boot environment still references their earlier blocks.

Measure the pool and dataset properties separately

Start with pool health and allocation:

zpool list
zpool status -x
bectl list -a
zfs list -o name,used,available,referenced,usedbysnapshots,usedbydataset,usedbychildren,usedbyrefreservation -r zroot/ROOT

USED is not equivalent to “files visible in this mount.” ZFS breaks space into snapshots, the dataset itself, children, and reservations. REFER reports data accessible by the dataset, while clones and snapshots can share blocks whose freeing depends on every reference.

Do not start by deleting package caches or log files repeatedly. If a snapshot holds the old blocks, that changes current visible data without releasing the expected pool allocation.

Identify origins, snapshots, and holds

List creation times, mount status, clone origins, and snapshot user references:

zfs get -r origin,creation,mounted,mountpoint zroot/ROOT
zfs list -t snapshot -o name,used,refer,creation -s creation -r zroot/ROOT
zfs holds -r zroot/ROOT

A clone depends on its origin snapshot. Holds can prevent snapshot deletion. Replication tools, periodic snapshot jobs, or administrative tags may own those holds. Removing a hold without coordinating retention can break incremental replication or legal preservation.

zfs destroy -nvp previews many destructive operations. Use it before a recursive snapshot or dataset removal, but still review every listed target; dry-run output proves command selection, not business approval.

Classify boot environments before pruning

For every environment, record active/next-boot status, creation reason, FreeBSD version, kernel/userland consistency, last successful boot, owner, and required retention. Preserve at least one verified known-good rollback before deleting old environments.

Use bectl to manage boot environments rather than destroying their datasets behind its back:

bectl list
bectl destroy old-upgrade-test

The command may warn about snapshots or dependent clones. Do not add force until you understand them. An environment can contain separate child datasets or snapshots created by other tools; the exact destroy scope belongs to the installed bectl(8) implementation.

If an environment is the active or next-boot target, activate a known-good alternative and confirm loader state before any reboot. A dataset that is mounted only temporarily for inspection must be unmounted cleanly.

Space may not return immediately or linearly

Deleting one snapshot frees only blocks not referenced by any other snapshot, clone, or live dataset. Adjacent snapshots commonly share most blocks, so their individual USED values are estimates of exclusive referenced space in context, not sizes that simply add like archive files.

Deferred destroy, pending background work, reservations, zvols, and pool slop space also affect visible availability. Wait for the supported operation to complete and re-measure properties. Never import a nearly full pool read-write on multiple hosts to “clean faster.”

Prevent recurrence with lifecycle metadata

Create boot environments with names that encode purpose rather than only a timestamp, and attach an owner/expiry in the change record. Automate inventory and alerts, not blind destruction. A cleanup job should refuse to delete active, next-boot, held, young, or unverified environments and should emit a dry-run report for review.

Before upgrades, forecast free space from realistic package/base deltas and current snapshot growth. Boot the new environment, validate services, then explicitly retire older generations according to policy. Boot environments accelerate software rollback; they do not replace an off-pool backup.

Close with a recovery test

After pruning, check pool health, available space, bectl list, loader selection, mounts, replication bases, and snapshot jobs. Reboot into the intended environment during a window and confirm kernel and userland versions match.

The repair is complete when space ownership is explained and a verified rollback remains—not when a force-delete command happens to raise the free-space number.

Related:

Sources: