Skip to content
FreeBSDDeep Dive Published Updated 8 min readViews unavailable

FreeBSD tmpfs in Production: Capacity, Memory Pressure, and Persistence

Operate FreeBSD tmpfs deliberately: understand its memory and swap accounting, size limits, inode pressure, boot persistence, monitoring, and safe rollback.

FreeBSD’s tmpfs is a volatile filesystem backed by virtual memory. It is useful for scratch data whose loss at unmount or reboot is acceptable, but “in memory” does not mean “free,” “unbounded,” or “guaranteed to remain in physical RAM.” A tmpfs mount competes with processes and the kernel for memory, can consume swap-backed pages under pressure, and can fail writes when its configured capacity or inode allowance is exhausted. Treat it as a separately budgeted resource rather than as a faster directory.

The important operational distinction is between a namespace and a persistence guarantee. A pathname such as /tmp looks like ordinary storage to an application, yet tmpfs content vanishes when the filesystem is unmounted or the machine reboots. Applications must be able to recreate caches, sockets, lock files, and temporary work after that event. Any data whose only copy matters must remain on persistent storage or be copied there before cleanup.

What is actually stored and reclaimed

The FreeBSD tmpfs implementation stores file metadata and file data in memory. File data may be paged to swap when the system is under memory pressure and swap is configured; tmpfs is therefore not a promise that every byte remains resident in DRAM. The current implementation keeps metadata, including directory contents, resident rather than swapping it out. A workload that creates millions of tiny files can use significant memory even when the combined file contents appear small. Estimate object count and directory scale, not just total bytes.

The mount’s size is a filesystem capacity limit, not a reservation of that amount of RAM. A mount configured with size=2g does not preallocate two gigabytes at mount time. Conversely, leaving the limit at its default does not make the space unlimited: the system-wide tmpfs policy caps use relative to memory plus swap, and the filesystem can refuse new files or extensions when the limit is reached. A process may then see ordinary filesystem errors such as ENOSPC even while df reports some free blocks, if the inode limit is the binding constraint.

This is why tmpfs should not be confused with a block device backed by md(4). tmpfs is a filesystem implementation with its own vnode and VM behavior. An md device exposes a block interface on which an administrator can place UFS or another filesystem. The two options serve different purposes and have separate capacity, accounting, and teardown paths.

Choose directories that can be rebuilt

Typical candidates include temporary build intermediates, short-lived decompression work, and an intentionally volatile /tmp. Avoid putting databases, package repositories, logs, swapfiles, or irreplaceable work there. Large compiler jobs, browser caches, and parallel test suites can exceed a conservative memory budget quickly. A system that previously wrote these files to a disk filesystem may change behavior significantly when the same path moves to tmpfs, because disk-backed capacity and VM pressure have different failure consequences.

Before mounting over an existing directory, inspect its current contents and consumers. Mounting hides the underlying directory entries without deleting them; unmounting reveals them again. A service that opened a file before the mount may continue using the underlying object while new path lookups see the tmpfs copy. This split view is a reason to make the change during a maintenance window and restart or quiesce users of the path before switching filesystems.

The following bounded test uses a temporary mountpoint rather than replacing /tmp. It requires root privileges and a kernel with tmpfs support. The 2 GiB value is an example cap, not a recommendation for every host:

install -d -m 0755 /mnt/tmpfs-lab
mount -t tmpfs -o size=2g tmpfs /mnt/tmpfs-lab
mount | grep '/mnt/tmpfs-lab'
df -h /mnt/tmpfs-lab
df -i /mnt/tmpfs-lab
swapinfo -h

The mount output verifies that the path is backed by tmpfs, while df and swapinfo provide different views of current filesystem capacity and host swap. These snapshots are not a forecast of peak demand. Exercise the actual application workload and record both peak file count and peak allocated data. A successful mount alone says nothing about whether the capacity is appropriate.

After confirming that no process is using the test mount, remove it in this order:

fstat -f /mnt/tmpfs-lab
umount /mnt/tmpfs-lab
rmdir /mnt/tmpfs-lab

If umount reports the filesystem is busy, identify open files and working directories and stop the owning service normally. Do not use a forced unmount to make a test appear successful. It can turn an orderly resource cleanup into application errors, and it does not recover data that the application expected to persist.

Bound capacity and inode use

A size option is useful for limiting the maximum amount of tmpfs data and metadata space charged to that mount. Choose the limit from measured peak demand plus a documented margin, while leaving room for the operating system and unrelated workloads. The global vfs.tmpfs.memory_percent policy applies to filesystems whose requested size is zero or otherwise uses the available-memory default. Review the running release’s tmpfs(4) page before changing this sysctl: changing a global default is not a substitute for understanding each mount’s explicit options.

The inode limit is separate from the byte limit. tmpfs can have space available yet fail to create another small file after its node allowance is consumed. Where the workload has a known high file count, test the real number and distribution of entries. The inodes mount option can cap nodes, but an arbitrarily high value is not harmless because each node requires metadata. Monitor both df -h and df -i and alert on exhaustion before application writes begin failing.

Disk usage, memory pressure, and swap use must be interpreted together. df reports filesystem-level allocated and available space. swapinfo reports swap devices and usage, not which individual directory generated every page. vmstat helps establish system-wide memory and paging trends but is not per-mount attribution. Correlate these measurements with workload timing, process memory, tmpfs capacity, and application logs. If system responsiveness deteriorates when the mount grows, the correct fix may be a smaller workload, concurrency limit, persistent disk-backed workspace, or a different system size, not a larger tmpfs cap.

Tmpfs data can be lost not only in a reboot but also in an operational unmount. A deployment script that mounts over a path, copies new files there, and then unmounts during rollback can discard the deployment state. Keep temporary data and configuration artifacts out of this lifecycle unless loss is intended and tested.

Make a mount persistent carefully

FreeBSD can mount tmpfs through /etc/fstab. The official manual’s minimal form is:

tmpfs   /tmp    tmpfs   rw,size=2g   0   0

The source string tmpfs is a conventional label for this filesystem; the type field selects the tmpfs implementation. The row makes the mount part of normal filesystem startup. Some systems also use tmpmfs-related rc.conf settings to manage temporary mounts. Do not configure both mechanisms for the same path without understanding their order and behavior. Inspect /etc/fstab and /etc/rc.conf first, and preserve local mount options rather than replacing a working row wholesale.

Changing /tmp affects every service that assumes temporary files are available at boot. Check boot-time programs, package scripts, daemons, cron jobs, and recovery procedures. Ensure the chosen capacity allows package tools or maintenance jobs to complete. A full tmpfs at boot can produce a failure far from the process that consumed it, such as a daemon unable to create a pid file or a package hook unable to stage data.

For a first activation, schedule console or out-of-band access, keep the prior fstab contents, and have an explicit rollback. A maintenance test can mount the row with mount /tmp after validating its source and options, then inspect mount and df output. To roll back, stop consumers, unmount cleanly, remove only the newly added entry, and restore the previous persistent configuration. Reboot acceptance should verify that the expected mount exists, its limits are correct, normal services start, and disposable data is recreated after restart.

Failure diagnosis and acceptance criteria

When a write fails, separate the main causes before changing limits. ENOSPC may mean the configured size is exhausted, the global tmpfs budget is reached, or the inode count is exhausted. EROFS or permission errors suggest mount mode or path ownership instead. A mount failure can result from a missing mountpoint, a filesystem type unavailable in the kernel, or a conflicting existing mount. Capture mount output, df, df -i, swapinfo, relevant sysctls, and the exact system log message before unmounting or remounting.

If a mount is unexpectedly absent after boot, compare the fstab row with the active mount table and logs, inspect whether noauto or another rc.conf mechanism controls it, and verify that the mountpoint exists before the consumer starts. Do not infer success from the existence of a tmpfs line in a config file. The runtime mount table is the evidence that the filesystem is active.

The change is ready for production only when representative workloads fit within the mount and inode budgets, memory and swap remain acceptable under concurrent load, restart behavior is tested, and services recreate all required ephemeral state. Keep a persistent fallback for data that must survive, alert on byte and inode pressure, and document an operator procedure for a busy mount. These checks make tmpfs a deliberate volatile tier rather than an accidental single point of data loss.

Related:

Sources:

Comments