Understanding GEOM: FreeBSD's Modular Storage Framework

Every time FreeBSD partitions a disk, exposes a persistent label, mirrors providers, or applies GELI block encryption, it is using GEOM. GEOM is the kernel’s modular disk-transformation framework: classes attach to existing storage and may expose new storage to the next layer. The result is a live topology rather than a flat list of /dev names.

That topology is powerful and unforgiving. A command aimed at da1 acts at a different layer from the same command aimed at da1p2, gpt/data, mirror/data, or gpt/data.eli. Before changing storage, identify which node is the physical provider, which transformations sit above it, and which consumer currently holds it open. Device names alone do not express the full dependency graph.

Providers and consumers

A provider offers a block-storage interface. A raw disk is a provider, a GPT partition is another provider, and a completed mirror exposes another. A consumer attaches to a provider and requests access modes such as read, write, or exclusive access. A geom is an instance of a GEOM class connecting consumers and providers. Classes such as PART, MIRROR, LABEL, and ELI implement transformation or organization rules.

A class can consume one or more providers and create a new provider. The filesystem above it does not need to understand how the lower layer implements redundancy or encryption. This contract is why stacking works, but it does not mean every possible order is equally safe or maintainable.

# Inspect classes and providers without changing them
geom -t
geom disk list
geom part list
gpart show
geom mirror list
gstat

geom <class> list shows configuration and relationships for that class. gpart show presents partition geometry, while gstat reports live I/O statistics and queue behavior. Record this evidence before detaching or retasting anything. A provider absent from one class may still be busy through another layer.

One possible UFS-oriented stack encrypts a mirrored provider:

UFS  →  geli (encryption)  →  gmirror (RAID1)  →  ada0, ada1

Arrows here mean “consumes the provider to the right.” Reversing the mirror and encryption layers changes failure handling, metadata placement, and how many keys or encrypted providers are managed. ZFS has its own redundancy, checksumming, volume management, and native encryption options; blindly stacking GEOM RAID below ZFS can hide individual devices and interfere with the failure visibility ZFS expects. Architecture should be chosen deliberately rather than because every layer creates a device node that accepts zpool create.

Tasting, metadata, and access conflicts

GEOM classes can taste providers as they appear, inspecting metadata to determine whether the class should attach. This is how a mirror or label can reappear after reboot. Metadata placement matters: gmirror(8) stores metadata in the last sector, and GPT also maintains backup metadata at the end of a disk. The FreeBSD Handbook therefore warns against mirroring entire GPT disks with metadata-based gmirror; common designs mirror partitions or otherwise ensure metadata schemes do not collide.

An existing consumer can prevent a topology change. Mounts, swap, an imported ZFS pool, or another GEOM class may hold read, write, or exclusive access. Forcing a detach because a provider is “busy” can turn a protective refusal into corruption. Use mount, swapinfo, zpool status, geom ... list, and fstat to identify consumers instead.

GPT partitioning: gpart

gpart(8) manages partitioning schemes through GEOM PART. The following destroys prior partitioning state and is appropriate only for an explicitly disposable laboratory disk:

gpart create -s gpt da1
gpart add -a 1M -t freebsd-boot -s 512K da1
gpart add -a 1M -t freebsd-zfs -l labdata da1
gpart show -p da1

The -a 1M alignment request avoids beginning a partition at an unsuitable physical boundary on common storage. The -l label creates a stable provider below /dev/gpt/labdata, while the same partition also has a kernel name such as /dev/da1p2. Stable labels are preferable in /etc/fstab because probe order can change, but labels must be unique. Check both GPT and generic GEOM labels with:

gpart show -l da1
glabel status
ls -l /dev/gpt

Before destructive work, gpart backup da1 produces a textual partition-table backup. It does not back up filesystems, GELI metadata, mirror metadata, or user data. Recovery planning must cover each layer.

Software RAID: gmirror, gstripe, graid3

gmirror implements RAID1, gstripe implements RAID0, and graid3 implements byte-level striping with a dedicated parity disk. They share GEOM concepts but not identical failure properties. RAID0 loses the whole volume when one component fails; RAID1 can continue from a surviving component; neither is a backup.

Creating a mirror writes GEOM metadata and should use unused providers whose contents are already backed up:

gmirror load
gmirror label -v data /dev/da1 /dev/da2
gmirror status
geom mirror list data

# Planned removal of a still-present component, then insertion of a replacement
gmirror remove data da2
gmirror insert data /dev/da3
gmirror status

gmirror remove applies to a component that is still present; an already disconnected component requires a different recovery decision. gmirror forget is not a generic “simulate failure” command—it forgets disconnected components. Using it as a test can discard information needed for recovery. A real replacement procedure starts from gmirror status and geom mirror list, confirms serial numbers and provider sizes, chooses the command matching the component’s actual state, inserts a suitable replacement, and watches synchronization to completion.

The resulting /dev/mirror/data behaves as a block provider, but “accepts a command” does not mean “is a recommended architecture.” Filesystem creation destroys previous contents. dd can bypass filesystem protections. ZFS generally benefits from direct knowledge of its component disks. The layer above GEOM must still be designed with its own recovery and integrity model.

Encryption: geli

geli(8) provides sector-level encryption for a GEOM provider. Initialization writes metadata and can make existing data inaccessible; the target and key strategy must be decided before running it. On a disposable partition labeled secure, a basic interactive flow is:

geli init -s 4096 /dev/gpt/secure
geli attach /dev/gpt/secure
newfs -U /dev/gpt/secure.eli
mount /dev/gpt/secure.eli /mnt

The .eli provider exists only while attached. Keyfiles, passphrases, metadata backups, boot-time attachment, and remote recovery all require an explicit plan; losing the required key material is data loss by design. A GELI layer protects blocks at rest but does not authenticate logged-in applications, conceal data after attachment, or replace backups.

ZFS native encryption works at the dataset level and has different replication and key-management semantics. GELI remains useful when encryption must cover a block provider independent of the filesystem. Choose based on the required security boundary, not on the assumption that two features named “encryption” are interchangeable.

Journaling: gjournal

GEOM also includes classes for labels, concatenation, striping, remote access through GEOM Gate, and legacy journaling designs. The current Handbook still documents UFS journaling through GEOM, but a new installation should compare that design with UFS soft updates journaling and with ZFS rather than layering gjournal from an old recipe. Filesystem and release documentation, not historical familiarity, should drive the choice.

GEOM Gate can export a provider to a client, but the Handbook warns that its network connection is not encrypted and that a filesystem must not be mounted concurrently by multiple ordinary clients. A remotely reachable block device is not a clustered filesystem.

Diagnosing a stacked provider safely

Start at the top-visible symptom and walk downward. If /dev/gpt/secure.eli is missing, confirm whether /dev/gpt/secure exists before reinitializing anything. If the GPT label is missing, inspect gpart show -l and geom part list. If the partition is missing, confirm the physical disk and its serial number with geom disk list. Recreating metadata before discovering why a lower provider disappeared can overwrite the evidence needed to restore it.

For performance, use gstat to identify which providers carry latency and queue depth. Its -p option deliberately limits display to physical rank-1 providers, so omit it when walking a complete layered graph. A busy encrypted provider and a busy component disk are two observations of the same I/O at different graph layers; do not sum them as independent workload. Align partitions, consider the providers’ reported sector sizes, and benchmark the complete stack.

For maintenance, record the intended topology in documentation and monitoring. Alert on degraded mirrors, missing components, resynchronization, and provider-name changes. A graph that reconstructed automatically after reboot still may have reconstructed in a degraded state.

Why the layered model matters

GEOM’s uniform provider/consumer model makes storage components composable and observable. It does not make composition consequence-free. Metadata locations, access conflicts, recovery order, alignment, encryption keys, and the expectations of the filesystem above remain the administrator’s responsibility.

The safest operational habit is to think in graphs: name the provider being changed, list its consumers, identify the class that created it, and predict what provider should appear afterward. If that explanation is unclear, the next command should be an inspection command rather than a write command.

Related:

Sources: