FreeBSD gmirror Operations: Build, Monitor, Replace, and Recover Mirrors

gmirror is FreeBSD’s GEOM class for maintaining mirrored block providers. A mirror can keep a volume available after a member device fails, but it does not protect against accidental deletion, filesystem corruption copied to both members, theft, fire, controller failure shared by both disks, or an administrator selecting the wrong provider. Treat it as one layer of availability, not a backup.

The commands here use data disks named da0 and da1 only as examples. Device names are not identities. Before any label, insert, remove, destroy, or clear operation, confirm the physical device by serial number, enclosure slot, and capacity. Commands that write GEOM metadata can destroy access to existing data.

Map the devices and the intended topology

Start with a read-only inventory:

camcontrol devlist
geom disk list
gpart show -lp
geom mirror status

Record model, serial, size, partition layout, and current mounts. Confirm that the two members are separate physical failure domains where possible: different drive slots alone do not protect against a common controller, power supply, cable bundle, or enclosure failure. A mirror’s usable capacity is constrained by the smaller member, and the mirror is only as available as the devices and path that feed it.

If either provider contains data, stop and make an independent backup before proceeding. A mirror label stores metadata in the provider’s last sector. Even though this is usually a small region, it is a write and a topology change. Do not assume that adding a blank member later will preserve data on a disk that is currently mounted or has a partition map.

Create a mirror for new data

The simplest controlled example is a mirror of two blank whole-disk providers. First check that da0 and da1 are truly the intended empty devices. Then create the mirror:

gmirror label -v data da0 da1
gmirror status
gmirror list

The resulting GEOM provider is usually available as /dev/mirror/data. Validate the provider and its component state before creating a filesystem:

geom mirror list data
geom disk list da0 da1

For a whole-device UFS filesystem, an illustrative sequence is:

newfs /dev/mirror/data
mkdir -p /srv/data
mount /dev/mirror/data /srv/data
df -h /srv/data

This intentionally simple layout is not appropriate for every production system. If partitioning, boot support, labels, swap, or encrypted layers are required, plan the GEOM stack and bootloader path before writing. A filesystem made on /dev/mirror/data is not automatically a bootable root configuration. Follow the Handbook and boot-layout documentation for the platform and firmware mode.

The geom_mirror kernel module can be loaded for an immediate test with kldload geom_mirror. For persistent loading, gmirror(8) documents geom_mirror_load=“YES” in /boot/loader.conf. Ensure the module is available early enough for any filesystem required during boot. Test from a console or a recoverable boot environment before placing root storage on a new topology.

Add a member without overwriting an existing data source

For migration, a common pattern is to label one existing provider as the initial member, then insert a new blank provider for synchronization. This is not an automatic data-migration guarantee: the current source must be authoritative, the new destination must be identified correctly, and the resulting mirror must be observed until synchronization finishes.

The upstream manual’s example is:

gmirror label -v data da0
gmirror insert data da1
gmirror status

Use that only when da0 contains the intended valid source data and da1 is the intended destination. The mirror metadata and synchronization direction matter. Verify the chosen component and mirror state with gmirror list before mounting or writing through the mirror. Never insert a disk that may contain the only current copy of data and expect gmirror to merge filesystems; it mirrors blocks, not directories.

On a mirror with autosynchronization disabled, explicit rebuild is needed after inserting the replacement. Otherwise, the class manages stale member synchronization automatically. Follow the local gmirror manual and inspect status rather than repeating rebuild commands speculatively.

Monitor normal and degraded operation

Use gmirror status for a concise synchronization view and geom mirror list for detailed component state. Also observe the underlying device and kernel logs:

gmirror status
geom mirror list
dmesg | tail -100
iostat -x -w 1

The synchronization status is not the same as application health. A mirror can be synchronized while a filesystem is full, an application is failing, or both members are returning slow reads. Establish baseline latency and error counters before an incident. Compare read/write errors, queueing, and kernel messages against the physical component path.

If a mirror becomes degraded, preserve its metadata and capture status before making changes:

gmirror list
gmirror dump /dev/da0
geom disk list da0

Use the actual provider path shown by the system. Do not run clear, destroy, or forget as a generic repair action. clear removes mirror metadata from a component, destroy clears metadata from components, and forget discards records for disconnected components. Those commands can eliminate evidence or make a valid component harder to reattach.

Replace a failed member methodically

First determine whether the member actually failed or only lost connectivity. Check drive telemetry, cable and backplane state, controller logs, and the operating system’s provider list. If the same device may return, preserve the state and consult the documented reattachment procedure. If the hardware is failed, obtain a replacement whose usable sector count is at least sufficient for the mirror’s recorded size.

Before replacement, record serial numbers and a complete status snapshot. If the failed component is permanently gone, gmirror forget may be needed to discard the disconnected member record. Then insert the new provider and watch synchronization. Exact ordering depends on whether the member is present, inactive, or failed; follow the gmirror(8) examples for that specific state and do not copy a command from a different failure mode.

After insertion, wait for synchronization to complete and verify:

gmirror status
geom mirror list data
gpart show /dev/mirror/data
fsck_ffs -n /dev/mirror/data

The last command is appropriate only for a UFS filesystem and only when the filesystem is unmounted; substitute a filesystem-specific read-only check where applicable. A successful block mirror rebuild does not by itself prove application consistency. Confirm services, recent backups, and a documented restore path.

Failure domains and performance tradeoffs

Mirroring writes each block to all active members, so writes must be accepted across the mirror’s required members. Read balancing can use different policies; gmirror(8) documents load, prefer, round-robin, and split algorithms. The default is load balancing. A read policy change is not a generic speed fix: workload size, latency, device characteristics, cache behavior, and synchronization requirements influence the outcome.

A mirror does not provide independent transactional application copies. For databases, keep database-native backup and recovery mechanisms. For boot storage, understand how each member contains bootcode and partition metadata; the filesystem provider alone may not make an alternate drive independently bootable. Test booting from the surviving member as a planned exercise, not during a live failure.

Plan rebuild capacity and duration before you need them. A full synchronization reads from a surviving source and writes a replacement, competing with the production workload for device service time. Large volumes can take hours or longer, and gmirror does not make the mirror healthy merely because a replacement disk has been inserted. Record expected throughput, maintenance windows, and alert thresholds; watch for the synchronization counter to advance and for kernel errors to remain absent.

The first member used to create a mirror is operationally significant when gmirror must decide which copy to trust after an unclean shutdown. The manual documents component priority and synchronization behavior. Avoid changing balance or priority settings during an incident unless you understand the source-selection consequences. If two members may have diverged, do not declare one authoritative by guessing; preserve both and use the documented mirror metadata and recovery process.

Also distinguish member failure from a filesystem inconsistency. GEOM can successfully expose a mirror while UFS still needs a clean unmount or fsck. Conversely, a filesystem checker cannot repair a failed disk transport. Diagnose at the layer reporting the error, and avoid running simultaneous repair commands against the same provider.

Acceptance and change records

A gmirror deployment is complete when the correct providers are present, expected filesystem layers are mounted above the mirror, synchronization is finished, kernel and device logs are clean, monitoring alerts on member loss, and recovery has been rehearsed. Capture a before/after topology, serial-to-slot mapping, member health, and rebuild duration.

After a replacement, keep an observation window long enough to notice recurring transport errors or repeated disconnects. Do not clear error counters until you have recorded them and the underlying cause has been investigated. Keep a separate backup on another system or medium with independent failure characteristics.

Related:

Sources: