FreeBSD GEOM I/O Observability: Read gstat and iostat Without Double Counting

FreeBSD storage performance is visible at several layers. GEOM presents providers and consumers through transformations such as partitioning, mirrors, encryption, and filesystem access. gstat reports transactions on GEOM devices, while iostat reports kernel device statistics and CPU or terminal activity. These tools answer related but different questions.

The central diagnostic mistake is to add every layer’s bytes together as if each were independent physical traffic. A write may be visible at a filesystem provider, a mirror provider, and one or more physical disks. That is useful topology evidence, but summing these rows can count the same logical work multiple times. Start from a hypothesis, identify the layer, and compare matching time intervals.

Inventory the GEOM path before sampling

Record the filesystem mount, provider stack, pool or mirror status, and physical devices:

mount
geom disk list
geom part list
geom mirror status
zpool status -v

Only run the commands relevant to the host. A ZFS pool may not have a mounted UFS provider at the same point in its path; a GEOM mirror may expose the provider consumed by a filesystem. Use geom list or class-specific commands when the chain is not obvious. Match device identifiers and labels to physical hardware rather than assuming the device unit number is stable.

Write down a topology such as:

filesystem -> mirror provider -> ada0 and ada1

or:

ZFS dataset -> pool vdev -> physical providers

This map prevents attributing a pool-level queue to the wrong physical drive. Also capture baseline service latency, host load, and workload context. A one-minute disk sample during idle does not explain a ten-second application stall from an earlier period.

Use gstat for GEOM transaction visibility

Run gstat interactively at a readable interval:

gstat -I 1s

The default display focuses on GEOM producers. Press c to toggle consumers, or start with -c if you need both sides of transformations. Use -p to show only physical providers. A filter can narrow the display:

gstat -p -f 'ada[0-3]$' -I 1s

The regular expression applies to device names. Quote it so the shell does not expand square brackets. This example selects providers ada0 through ada3; adjust the expression to actual names. Do not use a broad filter that quietly excludes the layer being investigated.

Batch mode collects a sample and exits when output is not a terminal:

gstat -b -f 'ada[0-3]$'

For a continuous CSV stream, gstat -C implies endless batch mode; send it to a monitored collector with timestamps and a stop condition. Avoid leaving an unbounded command writing to a filesystem that is itself under pressure. Restrict retention and record the collection interval.

gstat shows transactions at GEOM providers and consumers. Its busy percentage is not a universal disk utilization guarantee, and its output should be interpreted with queue depth, latency, request size, and physical device behavior. A provider can be busy because of queued work even when application throughput is modest. Compare the same device over the same interval before and during the event.

Use iostat for device rates and latency signals

iostat gives a system-level device summary and supports extended disk statistics:

iostat -x -w 1 -c 10

With -x, the output includes read and write operations per second, throughput, queue length, average transaction durations, and busy time when available. The first output is generally averaged over system uptime; later samples are averaged over each requested interval. Use repeated samples for current behavior rather than interpreting the first long-term average as an incident measurement.

To focus on specific device names:

iostat -x -w 1 -c 10 ada0 ada1

Use names that exist on the host. If a display column is omitted because of terminal width, pass a device list or adjust output collection. The -z option can omit idle devices with extended statistics. Be careful when comparing results from different flags: -I changes rates into total values for the time period, so it changes the meaning of several fields.

Queue length and transaction time are clues, not root-cause labels. High average time can come from the device, controller, transport, contention, or workload shape. A low throughput result may indicate small random operations, throttling, or a stalled queue rather than a simple bandwidth ceiling.

Correlate storage layers without double counting

Collect gstat and iostat with aligned intervals, then correlate timestamps with application logs and kernel messages. If activity is high on a filesystem-facing provider but absent on physical disks, investigate caching, the selected layer, or a measurement mismatch. If the physical provider is busy while a mirror provider is not, compare the time windows and determine whether unrelated I/O is reaching that member.

For a mirror, a write must be handled by required active members, and a slow member can influence completion latency. Observe the mirror provider and each underlying disk. For ZFS, use zpool iostat -v to inspect pool/vdev logical throughput and latency details, then use physical device statistics as a separate layer. Do not add ZFS logical bytes to GEOM physical bytes to claim total network-like throughput.

For encryption or partition stacks, the same I/O can pass through multiple providers. Use the topology map and choose one logical layer for application-facing rates and one physical layer for device behavior. State which layer each graph measures. This is essential when comparing monitoring systems that label a mirror or encrypted provider differently.

Build a repeatable incident sample

For a user-visible stall, gather a short sample while the problem occurs:

date
uptime
iostat -x -w 1 -c 10 ada0 ada1
gstat -p -I 1s

Run interactive commands in separate terminals if needed and note their exact start times. Do not run a high-volume benchmark during the incident before collecting passive evidence; a benchmark changes queueing and cache state. If a controlled synthetic workload is necessary, schedule it separately and use a test filesystem or device.

Capture dmesg and relevant service logs at the same time. CAM resets, transport timeouts, device detach/attach, filesystem errors, and ZFS checksum errors are more actionable than a throughput graph alone. Protect diagnostic files and avoid storing application data or credentials in a shared incident bundle.

Interpret patterns cautiously

Consistently high service time on a physical disk with modest queue depth can point toward internal retries, media behavior, or controller latency. High queue length and high busy time during a known batch can simply reflect the workload. Similar rates across mirror members but much higher transaction time on one member can make that member a bottleneck. Confirm with device logs and repeated samples before replacing hardware.

If gstat shows consumers, use them to follow a request through GEOM classes, not to count bytes. A consumer row is the relationship by which a class uses an upstream provider; it is not a second physical drive. A provider can appear at multiple points in the stack.

If the first iostat sample is dramatically different from later samples, that is often because it aggregates since boot. Compare interval samples. If one command says the system is idle and another says a device is busy, inspect the selected device set, sampling windows, and whether the tools are reporting a logical or physical layer.

Treat average transaction duration as an average, not a tail-latency percentile. A small number of slow operations can be hidden by a large count of fast ones, while a short sampling interval can exaggerate a transient. For latency-sensitive applications, correlate these kernel averages with application-side histograms and request deadlines. Keep clock synchronization and sample timestamps consistent before comparing data collected on different hosts.

The gstat display also depends on which providers are currently attached. A filter such as ada[0-3]$ does not include da devices, NVMe providers, mirror names, or a newly numbered disk outside the range. Re-run the inventory after a reboot, hotplug, replacement, or topology change. If a graph unexpectedly goes quiet, verify discovery before treating it as proof of an idle workload.

Acceptance criteria for monitoring

A storage dashboard is useful when each series names its layer, reports units and interval, identifies physical devices stably, and preserves enough context to correlate with service latency. Alert on sustained symptoms and error counters, not a single short busy spike. Keep baseline measurements for comparable workload windows.

After a hardware or topology change, refresh the device map and monitoring filters. Device unit numbering can change, and a graph attached to the old path may go quiet while the new device receives all traffic. Verify the monitor with a controlled, low-risk read/write test after a change window, then confirm that logs and dashboards show the expected layer.

Related:

Sources: