FreeBSD Kernel Memory Triage with vmstat, UMA, and malloc

Kernel memory investigations go wrong when “RAM used” is treated as one number. A FreeBSD host has user-process mappings, filesystem cache, ZFS ARC, kernel virtual memory, network buffers, and allocator-managed objects with different lifetimes and accounting. vmstat(8) exposes several useful views, but none is by itself a leak detector. The goal is to identify which accounting domain is growing, correlate that growth with workload and time, then collect evidence before changing limits or restarting the system.

This article focuses on kernel allocator observability: vmstat -m reports dynamic kernel memory associated with malloc types, while vmstat -z reports UMA zone use. They are separate views and their rows are not interchangeable. Use the installed release’s manual pages because allocator names and output can evolve.

Capture release and system-wide baselines

Start with the exact kernel and userland identity and a stable snapshot:

freebsd-version -kru
uname -a
vmstat -s
vmstat -m
vmstat -z
swapinfo -h

vmstat -s reports accumulated paging-related event totals since boot. The ordinary interval form reports rates for the interval after the first display; that initial display summarizes activity since boot. Keep this distinction in mind when comparing incident samples. A cumulative count that is high after months of uptime is not a current rate, and one interval can be noisy during startup or a backup.

Record when each sample was captured, uptime, workload, recent package/kernel changes, and whether the host is a ZFS system. Repeat the same read-only commands during normal load, the incident, and recovery. vmstat -m and vmstat -z produce snapshots; a time series is more informative than a single maximum. The name of a memory type or zone may reveal the subsystem to investigate, but it is a label, not proof of a defect.

Separate malloc types from UMA zones

The kernel malloc API associates allocations with a named malloc_type. Its manual says those statistics can be inspected with vmstat -m. Developers define a type so usage can be attributed and sanity-checked. This view is therefore useful when a driver or subsystem has a distinct type, but the summary is only as granular as the allocator labels available in that build.

UMA, the Universal Memory Allocator, manages collections of same-sized items called zones. The vmstat -z view reports memory used by the kernel zone allocator by zone. A zone can retain slabs or cached items to serve future allocations efficiently; retained allocator memory is not automatically a leak or reclaimable on the same schedule as an unused user process. Distinguish objects currently allocated from capacity held by allocator structures using the fields shown by the running system and the matching vmstat(8) / uma(9) documentation.

Use the two views as a classification step:

vmstat -m | less
vmstat -z | less

Do not add the reported values from -m and -z as if they were disjoint parts of physical RAM. They describe different allocation mechanisms and may not account for every kernel consumer or all metadata in the same way. Likewise, the sum of top rows is not a reliable substitute for total physical memory accounting. Use system-wide VM counters, process views, ARC statistics, and device-specific counters to reconcile the rest.

Read interval activity before blaming an allocator

Use explicit interval and count flags for a short live sample:

vmstat -w 1 -c 10
vmstat -w 5 -c 12

The command reports process/run-queue state, memory, paging rates, disk activity, trap rates, and CPU time. In the page section, pi and po are pages paged in and out per second; sr is pages scanned by the page daemon. Rising paging activity concurrent with latency is stronger evidence of current pressure than nonzero swap occupancy alone. swapinfo -h gives a useful capacity snapshot, but configured or previously used swap does not establish active page churn now.

avm is mapped virtual memory, not a direct count of resident physical pages. The vmstat(8) manual explicitly notes that it sums virtual pages belonging to mapped objects and does not represent the active page queue. fre is the size of the free list; a small free count alone does not establish pressure because the VM system can reclaim and reuse memory. Correlate rates, queueing, application latency, and system logs instead of tuning from one column.

For system-wide history, vmstat -s helps identify whether paging, faults, or other counters grew over the whole uptime. It cannot locate the process or allocator responsible for a change. For malloc/UMA attribution, compare -m and -z snapshots at known times. For process virtual mappings, use the existing process-inspection tools such as procstat -v <pid>; this measures a different layer and does not explain kernel zone allocations.

Build a reproducible allocator comparison

An operational baseline should include the same commands, the same interval, and the same workload labels each time. Save output with a timestamp and kernel identity without modifying kernel state:

date -u
freebsd-version -kru
vmstat -w 1 -c 10
vmstat -m
vmstat -z
vmstat -s
swapinfo -h

Repeat after the system has been idle long enough for transient buffers and temporary work to drain, then under the workload that triggers the issue. Compare the same allocator row across samples and inspect whether InUse, memory use, requests, or other fields grow together; field names and meanings should be taken from that release’s output and manual rather than assumed from a parser written for another version. A higher request count with stable in-use memory can mean churn, not retained growth. A larger high-water mark can record past peak allocation rather than current use.

Trend sampling should be rate-limited. Running high-frequency inventory commands forever can add noise and produce excessive monitoring data. If automated collection is needed, use a bounded script that records the command exit status, timestamp, kernel release, and raw output. Avoid brittle column-number parsing; allocator labels may contain spaces or change between releases, and formatted text is not a stable machine API. Where structured collection is important, investigate documented --libxo support on the installed vmstat and validate its schema before building alerts.

Investigate growth as a hypothesis, not a verdict

When one named malloc type or UMA zone rises during a controlled workload, ask:

Does the growth correlate with request rate, active connections, device events, or a known cache warm-up?
Does it plateau after the workload becomes steady, or continue growing while load and object count stay constant?
Does the same row shrink after the workload drains, after an allocator reclaim opportunity, or only after reboot?
Are page-in/page-out activity, free memory, kernel messages, and application latency changing at the same time?
Did the kernel, driver, firmware, module set, or system configuration change between the good and bad samples?

An increase in a zone can be expected when the system creates more sockets, vnodes, packet descriptors, or another workload-dependent object. A plateau at a new level can indicate a larger working set or a cache, not an unbounded leak. Continuous growth under a stable workload, allocator exhaustion messages, failed allocations, or a related panic provide stronger grounds for escalation. Preserve timestamps and the full rows; a screenshot of one vmstat invocation usually discards the context needed by a maintainer.

Do not respond by raising an unrelated memory tunable, limiting ARC, or rebooting before collecting a baseline. A VM cap can shift pressure to the application or change cache hit rates. Raising allocator limits can hide a leak while consuming memory needed by other subsystems. A reboot clears counters and state, which is useful for a controlled reproduction but destroys the evidence for an active incident.

Use diagnostic allocators only in controlled debugging

FreeBSD includes debugging facilities such as MemGuard for finding memory corruption in targeted malloc types or UMA zones. Its manual documents runtime and loader settings for selecting a type or zone, but guarded allocations can consume additional memory and alter allocator behavior. This is a kernel debugging tool, not a generic production leak monitor. Identify the exact malloc type or UMA zone from vmstat -m or vmstat -z, read the installed memguard(9) manual, and test the configuration in a disposable or redundant environment before applying it to a live workload.

Changing a loader-time setting can require reboot and may prevent a host from booting if the chosen debug configuration is too costly. Keep console access and a rollback boot entry. Runtime instrumentation can affect only later allocations for the selected type and does not retroactively guard objects that already exist. Build symbols and exact kernel provenance matter if a panic or diagnostic message must be interpreted afterward.

For suspected kernel defects, collect the output of vmstat -m, vmstat -z, interval samples, vmstat -s, kernel messages, kernel and module identities, and a minimal reproduction. Compare supported point updates and errata for the running release before attributing a row to a driver. If the system panics, retain the crash dump and matching symbols using the established crash-dump procedure; allocator output alone rarely identifies the corrupting call site.

Operational acceptance criteria

Close an investigation only when the baseline and incident windows are comparable, the suspected allocator is identified by supported evidence, and the remedy is validated under a representative workload. Record the row names, exact FreeBSD build, collection interval, traffic or workload, and whether the value returns to baseline. If a change modifies an allocator, driver, ARC ceiling, or kernel debug option, compare memory pressure, paging, application latency, and recovery behavior after reboot.

Useful alerts are based on sustained trends and user impact: repeated paging activity, allocation failures or kernel warnings, a zone that grows continuously under stable load, and a simultaneous latency or availability regression. Avoid alerts on “low free RAM” alone. FreeBSD’s memory system intentionally uses RAM for useful caches and allocator pools; evidence comes from whether the workload can allocate and make progress, not from maximizing one free-memory number.

Related:

Sources: