Fixing ZFS ARC Consuming All Available RAM on FreeBSD

ZFS uses otherwise idle RAM for its Adaptive Replacement Cache, so a host can show little “free” memory without being unhealthy. ARC is designed to shrink under pressure, but reclamation is not magic or always instantaneous: a mixed database, VM, jail, or build host can still swap, stall, or invoke the OOM path if its working sets and kernel allocations compete. Confirm ARC size and actual pressure over time before setting a cap.

Measure ARC and the VM system together

On FreeBSD 13 and later, the documented ARC limit sysctls use dots:

sysctl kstat.zfs.misc.arcstats.size
sysctl kstat.zfs.misc.arcstats.c
sysctl kstat.zfs.misc.arcstats.c_max
sysctl vfs.zfs.arc.max
sysctl vfs.zfs.arc.min

Older FreeBSD 12 used names such as vfs.zfs.arc_max; copying that spelling onto a modern host returns an unknown OID and changes nothing. Record freebsd-version -kru with every tuning report.

ARC size is only one memory consumer. ZFS also uses metadata and internal structures; deduplication tables, dirty data, vdev queues, and L2ARC metadata can matter. User processes, wired kernel allocations, network buffers, tmpfs, and jails compete outside ARC. Observe the system, not one number:

vmstat 1
swapinfo -h
top -o res
sysctl vm.stats.vm.v_free_count vm.stats.vm.v_inactive_count

Free-page counters are measured in pages and must be interpreted with VM thresholds and page size, not read as bytes. Look for sustained swap activity, rising paging, allocation stalls, killed processes, and latency in memory-hungry applications. A large ARC with zero swap I/O and healthy services is not evidence of a leak.

Test whether ARC relinquishes memory under a real workload

Capture ARC size, VM counters, swap, and application latency before and during the workload that triggers complaints. A database restart or VM boot should cause ARC’s target and size to respond while the application obtains memory. The response is asynchronous; expecting the ARC number to fall to a new limit in one sysctl refresh is unrealistic.

Do not manufacture production pressure with an uncontrolled allocator. Reproduce in staging or use the normal application workload during a maintenance window. If the system swaps heavily while ARC remains large, collect time-series evidence and check the current FreeBSD/OpenZFS errata before tuning.

Differentiate swap configured from swap actively used. Some swap occupancy can persist after pressure passes because FreeBSD has no reason to page cold memory back immediately. The diagnostic signal is ongoing page-in/page-out activity and workload latency, not a nonzero swapinfo line by itself.

Also inspect the applications. A JVM or database may reserve large virtual address space, have its own cache, or change allocation abruptly. An ARC cap cannot fix an unbounded process leak; it merely gives that leak more room.

Choose a ceiling from workload headroom

The FreeBSD Handbook says the default ARC maximum is chosen to use a large fraction of RAM and recommends a lower value when other daemons need memory. Calculate a ceiling from measured peak resident application memory, non-ARC kernel use, desired filesystem cache, and safety margin. Do not copy “4G” from a machine with a different workload.

Test a runtime maximum in bytes on a modern release:

sysctl vfs.zfs.arc.max=8589934592

That example is 8 GiB, not a recommendation. Confirm the accepted value and watch ARC shrink over time:

sysctl vfs.zfs.arc.max
sysctl kstat.zfs.misc.arcstats.size

If the cap restores application latency and removes active swapping without unacceptable storage I/O, persist the exact tested value in /etc/sysctl.conf or the release-documented loader configuration. Do not define the same setting in multiple files; later ambiguity makes incident response harder.

Avoid raising vfs.zfs.arc.min on a mixed-workload host without a specific reason. The minimum protects ARC from pressure and can reserve memory away from applications—the opposite of this repair. Ensure arc.min remains meaningfully below arc.max.

Measure the performance cost of a smaller ARC

ARC serves reads and metadata without storage I/O. A smaller ceiling can lower its hit ratio and increase disk latency, vdev load, and database or VM response time. Record ARC hit/miss counters and pool I/O before and after:

sysctl kstat.zfs.misc.arcstats.hits
sysctl kstat.zfs.misc.arcstats.misses
zpool iostat -v 5

Raw lifetime hit ratio can hide a recent regression, so collect deltas over the same representative workload. Metadata-heavy trees may suffer even when bulk read throughput looks unchanged. Revisit the cap after RAM, dataset, or workload changes.

Adding L2ARC is not free RAM. It can reduce repeated reads from primary storage, but it consumes RAM for metadata and does not serve as an application-memory reserve. Likewise, enabling deduplication can create substantial memory demand; compression is usually a different, less memory-intensive space-saving tool.

Investigate cases where a cap is not enough

If memory pressure continues after ARC stays below the verified cap, identify other wired or malloc consumers with vmstat -m, vmstat -z, process RSS, and kernel logs. Network mbufs, a driver leak, tmpfs, ZFS dirty data, or application caches require different controls.

On low-memory or 32-bit systems, address-space limits make ZFS tuning more sensitive. Follow the matching Handbook section rather than modern amd64 defaults. A pool’s capacity in terabytes alone does not determine the correct ARC size; active working set, dedup, metadata count, and I/O latency matter more.

Do not clear ARC repeatedly as routine maintenance. Cold-cache benchmarks and cache purges create artificial I/O storms and conceal whether normal reclamation works. A stable system should reach an equilibrium under sustained representative load.

Verify across reboot and peak load

After persisting the cap, reboot in a controlled window and verify the correct OID/value, ARC growth, swap activity, and application startup. Exercise the known peak workload, then compare latency and zpool iostat with the baseline. Alert on active swap I/O, OOM kills, and unexpectedly high non-ARC kernel memory—not simply “RAM used.”

The goal is not maximum free memory. It is enough ARC to reduce I/O while leaving reliable headroom for everything else. Evidence from both ZFS and FreeBSD VM counters is what distinguishes useful cache from genuine memory contention.

Related:

Sources: