ZFS on FreeBSD: Pools, Datasets, and Snapshots Explained
How FreeBSD ZFS pools, vdevs, datasets, snapshots, replication, scrubs, ARC, and boot environments behave in real operations.
ZFS combines a volume manager, a copy-on-write filesystem, checksums, caching, and replication primitives in one storage system. That integration removes several boundaries found in a conventional partition-plus-RAID-plus-filesystem stack, but it also makes topology decisions unusually consequential. A pool can contain many datasets, yet all of them ultimately depend on every top-level data vdev needed to reconstruct the pool.
FreeBSD ships ZFS as an integrated storage option and its installer can create a ZFS-root system. The implementation is developed with OpenZFS, while FreeBSD supplies operating-system integration including boot support, rc.d handling, GEOM device access, and boot environments. The running system’s zpool version and manual pages are the authoritative guide to available features; importing a pool with newer feature flags can make it unavailable to older rescue media.
Pools and vdevs define the failure domain
A pool aggregates one or more top-level virtual devices, or vdevs. A top-level data vdev can be a single device, mirror, or RAID-Z group. ZFS distributes allocations across top-level vdevs; redundancy is provided inside each redundant vdev. If the pool loses a required, non-redundant top-level vdev, datasets elsewhere in that pool are not independently safe merely because another vdev is a mirror.
The following syntax illustrates a mirror, but zpool create writes labels and can destroy access to existing data. Never paste it against real device names. Confirm backups, device serial numbers, partition labels, sector alignment, and the complete topology first:
# DESTRUCTIVE EXAMPLE — substitute verified, unused persistent labels only.
zpool create -o ashift=12 tank mirror \
/dev/gpt/tank0 /dev/gpt/tank1
zpool status -P tank
zpool get all tankPersistent GPT labels are preferable to transient probe-order names such as /dev/da0, especially when cables, controllers, or boot order change. ashift describes the sector-size exponent ZFS uses for allocation alignment; choosing it too small for the real media can impose lasting read-modify-write penalties. It is a creation-time topology decision, not a universal number to copy without checking the drives.
Adding a top-level vdev expands capacity and changes future allocation. It is not equivalent to attaching a disk to an existing mirror. Although modern OpenZFS supports removal only for certain topologies, administrators must not assume an arbitrary added RAID-Z or device can later be detached. Treat zpool add as a planned, backup-protected architecture change. Use zpool attach when the documented operation is specifically converting or extending a mirror, and inspect the exact proposed command before executing it.
ZFS redundancy also does not replace hardware monitoring. Checksums can detect corruption and mirrors or parity can repair it when a valid replica exists, but a pool cannot reconstruct data if all relevant copies are lost. zpool status reports ZFS-visible health; drive firmware health, controller errors, enclosure power, and backup restorability require separate monitoring.
Datasets share space but isolate policy
A filesystem dataset draws from the pool’s common free space and has its own mount point and properties. A volume dataset, or zvol, exposes a block device instead. Datasets are inexpensive administrative boundaries, so create them where snapshot, replication, quota, delegation, mount, or security policy differs—not merely to imitate every traditional directory.
zfs create tank/services
zfs create -o compression=lz4 tank/services/logs
zfs set atime=off tank/services/logs
zfs get -s local,inherited compression,atime,mountpoint tank/services/logsProperties normally inherit from parents. zfs get reports both the value and its source, which is essential when a setting appears to “change itself” after an ancestor is modified. zfs inherit removes a local override; it does not copy the parent’s current value permanently.
Compression is applied to newly written blocks. Enabling it does not retroactively recompress existing data, and changing algorithms affects subsequent writes. lz4 is a conservative general-purpose choice; zstd offers configurable tradeoffs. Claims that compression always improves performance are too broad: compressible workloads often reduce physical I/O, while incompressible or CPU-bound workloads may not benefit. Measure with representative data.
Capacity controls also have distinct meanings. quota limits a dataset and its descendants, while refquota limits the space referenced by that dataset itself. reservation and refreservation guarantee space and therefore reduce what appears available elsewhere. Snapshots and descendants can make used differ substantially from the size visible by walking the mounted files. Audit with explicit columns:
zfs list -o name,used,avail,refer,mountpoint
zfs list -t filesystem,volume,snapshot -o space
zfs get quota,refquota,reservation,refreservation tank/servicesSnapshots are consistent references, not free backups
A snapshot is a read-only view of a dataset at a transaction-group boundary. Copy-on-write lets ZFS create it quickly because existing blocks are referenced rather than copied. The snapshot consumes metadata immediately and retains old blocks as the live dataset changes. It is therefore cheap at creation, not “free.” A busy dataset with high churn can make a long-lived snapshot consume substantial pool space.
zfs snapshot tank/services@before-change
zfs list -t snapshot -o name,used,refer,creation
zfs diff tank/services@before-changeA recursive snapshot can capture a dataset hierarchy, but application consistency is a separate concern. ZFS makes the on-disk snapshot transactionally consistent; it cannot guarantee that an application has flushed its own buffers or that a multi-host transaction is logically complete. Databases may require an application-native checkpoint, quiesce hook, or coordinated backup procedure.
zfs rollback is destructive: it discards live changes after the target snapshot and may require destroying newer snapshots or clones depending on options. Before rollback, list dependencies, preserve anything needed, and prefer restoring selected files or cloning the snapshot for inspection when uncertainty exists.
A clone is a writable dataset initially sharing blocks with a snapshot:
zfs clone tank/services@before-change tank/services-test
zfs get origin tank/services-testThe origin snapshot cannot be destroyed while dependent clones remain. zfs promote changes dependency relationships; it is not just a rename and should be planned with snapshot naming and replication in mind.
Snapshots are not independent backups. Pool loss, mistaken recursive destruction, stolen hardware, or a privileged attacker can remove both live data and local snapshots. A backup needs another failure domain and a tested restoration path.
Send and receive replicate snapshot history
zfs send emits a stream representing a snapshot, and zfs receive reconstructs it as a dataset. An incremental stream describes changes between related snapshots. This preserves ZFS structure more directly than a file-tree copy, but performance depends on changed data, network, compression, encryption, and destination behavior; it is not universally faster than rsync.
# First replicate to a separately protected backup pool.
zfs send tank/services@week-01 | zfs receive -u backup/services
# Then send the incremental relationship.
zfs send -i tank/services@week-01 tank/services@week-02 | \
zfs receive -u backup/servicesThe source and destination names above are examples. zfs receive can overwrite or roll back data with force-related options, so preview the destination and avoid broad flags until their exact effects are understood. Resume tokens can continue interrupted receives on supported streams. Encrypted datasets add choices between raw and decrypted send behavior; consult the matching zfs-send(8) manual before designing key custody and off-site replication.
Verify replication by importing or mounting the destination in a recovery test, checking expected snapshots and properties, and restoring representative files. A successful pipeline exit proves that bytes moved, not that retention policy, keys, bootability, or application recovery is correct.
Scrubs, resilvers, and health answer different questions
A scrub reads allocated data and metadata, verifies checksums, and repairs damage when a good redundant copy exists. It does not scan unused sectors, replace SMART monitoring, or create redundancy. A resilver reconstructs data onto a replaced or newly attached device; it is driven by topology change and is not the same maintenance operation.
zpool status -x
zpool status -v tank
zpool scrub tank
zpool iostat -v tank 5
zpool history tankMonitor a scrub through zpool status and investigate checksum, read, or write errors instead of clearing them to make the display green. zpool clear resets recorded errors after the underlying fault is understood; it does not repair a cable, drive, controller, or corrupted replica. Schedule scrubs at a cadence suited to data importance and workload, then alert on completion and errors.
Keep pool capacity headroom. Copy-on-write needs free regions to allocate new versions before old versions can be released, and fragmentation plus very high occupancy can degrade performance. There is no single safe percentage for every topology and workload, so trend capacity, allocation behavior, snapshot growth, reservations, and workload latency rather than waiting for ENOSPC.
ARC and FreeBSD boot environments
The Adaptive Replacement Cache stores frequently useful ZFS data and metadata in RAM. High ARC use is expected and should not be labeled a leak from one top snapshot. On modern FreeBSD the documented upper-limit sysctl is vfs.zfs.arc.max—with a dot before max; older releases used vfs.zfs.arc_max. An upper bound limits ARC growth, while memory-pressure behavior is handled by the implementation. Tune only after measuring an actual contention problem, because shrinking cache can trade RAM headroom for substantially more I/O.
sysctl vfs.zfs.arc.max
sysctl kstat.zfs.misc.arcstats.size
sysctl kstat.zfs.misc.arcstats.hits
sysctl kstat.zfs.misc.arcstats.missesOn a ZFS-root FreeBSD installation, bectl(8) manages boot environments—bootable clones of the root environment. They are particularly useful before base-system upgrades:
bectl list
bectl create before-upgrade
bectl listIf an upgrade fails, bectl activate before-upgrade selects that preserved environment for the next boot. Confirm the desired environment name and activation result before rebooting. Boot environments improve rollback speed but share the same pool, so they are not backups either.
A sound ZFS operating model has four independent disciplines: design vdev redundancy before pool creation, use datasets to express policy, monitor and scrub the live pool, and replicate tested snapshots to another failure domain. ZFS provides unusually strong primitives, but it cannot compensate for an irreversible topology command, a single physical enclosure presented as “redundant,” or backups nobody has restored.
Related:
Sources: