FreeBSD procstat: Process Inspection Beyond ps and fstat

ps answers the first questions about a process: which PID it has, who owns it, and how much CPU or memory it appears to use. FreeBSD’s procstat goes further. It exposes several views of a live process through one interface, including open descriptors, virtual memory mappings, credentials, signals, threads, kernel stacks, and binary information. It can also read selected information from a process core file.

That breadth makes procstat useful when a service is alive but behaving incorrectly. Instead of immediately attaching a debugger or restarting the process, you can first record what the kernel knows about it.

Start with a specific PID

Run each investigation against an explicit PID whenever possible. procstat -a is convenient for an inventory, but a focused command is easier to preserve and compare.

pid=$(pgrep -xo daemon_name) || exit 1
procstat basic "$pid"
procstat binary "$pid"
procstat arguments "$pid"

The basic view resembles an expanded process listing. binary reports the executable and ABI details, while arguments records the actual argument vector. Use procstat environment with care: environment variables routinely contain tokens, proxy credentials, database URLs, and other secrets. Capture that output only when it is necessary, restrict the resulting file, and remove it after the incident.

The compatibility commands pargs, penv, and pwdx provide argument, environment, and working-directory views for administrators familiar with the Solaris utilities. They are interfaces to the same process-information machinery, not separate sources of truth.

Trace files, sockets, and capabilities

An unexpected file lock, deleted configuration file, or listening socket is often visible in the descriptor table.

procstat files "$pid"
procstat files -C "$pid"

The files view identifies descriptor numbers and object types such as vnodes, sockets, pipes, and kqueues. The capability column is especially important for a Capsicum-aware process because it shows which operations remain authorized through each descriptor. A pathname may be absent even when the descriptor is valid: procstat relies on the kernel name cache for some paths, and not every filesystem supplies a recoverable name.

Do not confuse an open descriptor with a current pathname. A process can keep using an inode after its directory entry has been renamed or removed. Compare the descriptor output with fstat, sockstat, and the relevant filesystem state before concluding that the application opened the wrong file.

Read the virtual address map

The virtual-memory view connects loaded objects and anonymous regions to address ranges and protection flags.

procstat vm "$pid"
procstat auxv "$pid"

This is useful when verifying which shared library was actually mapped, whether a file-backed region is executable, or how much anonymous memory a process accumulated. It is not a heap profiler: a large anonymous mapping does not identify the allocator object responsible for its contents. Use the map to narrow the question, then choose DTrace, allocator diagnostics, or a debugger for allocation-level evidence.

auxv displays the ELF auxiliary vector supplied at process startup. It can confirm details such as page size, program headers, and entry information when diagnosing loader or ABI problems.

Inspect threads, signals, and kernel stacks

A process that appears idle may have one thread blocked in I/O while another is waiting on a lock. Record its thread and signal state before changing anything.

procstat threads "$pid"
procstat signals "$pid"
procstat kstack "$pid"

Kernel-stack output normally requires sufficient privilege. Repeating it at a short interval can reveal whether a thread is making progress or waiting in the same kernel path. A stable stack is evidence, not a diagnosis by itself: sleeping in kevent, accept, or a condition variable may be normal for that thread.

The signal views distinguish dispositions from pending or blocked signals. This matters when a daemon ignores a reload, a supervisor reports a stop timeout, or a signal is delivered to a multithreaded program but never handled as expected.

Preserve machine-readable output

procstat supports libxo output. For incident collection, JSON or XML is safer to parse than columns intended for a terminal.

procstat --libxo json files "$pid" > "procstat-files-${pid}.json"

Record the FreeBSD version, process start time, PID, command, and collection timestamp alongside the output. PIDs are reusable, so a bare number without timing evidence can later refer to a completely different process.

Core-file support is also version-sensitive. A core produced by a different major FreeBSD version can have incompatible kernel structures. Keep the executable, debugging symbols, libraries, core, and OS version together if the investigation must survive a system upgrade.

Correlate limits, credentials, and scheduling state

Many incidents that look like application bugs are changes in execution context. Capture the process credentials, resource limits, current usage, CPU set, and thread state in the same snapshot:

procstat credentials "$pid"
procstat rlimit "$pid"
procstat rlimitusage "$pid"
procstat cpuset "$pid"
procstat threads "$pid"

The credential view includes real, effective, and saved user and group IDs, the umask, supplementary groups, and whether the process entered capability mode. That can explain why a daemon reads one file during startup but cannot reopen it after dropping privileges. The limit and usage views distinguish a configured ceiling from present consumption, while the cpuset and thread views show whether a process was confined to unexpected CPUs or is waiting on one channel.

Some views require the process owner or superuser, and kernel stacks require a kernel built with the relevant stack support. Treat an empty or unavailable field as missing evidence, not proof that the object does not exist. Record command exit status and stderr with the snapshot so a later reviewer can distinguish “none” from “not permitted.”

For a changing process, use -w only when repeated terminal output is useful. For durable evidence, collect separate timestamped libxo documents. That prevents a long incident log from mixing several process generations under a reused PID and lets comparison tooling report the exact descriptor, limit, credential, or thread field that changed.

procstat is most valuable as a disciplined first snapshot. It shows the kernel’s view without requiring a process restart, and it tells you which deeper tool is justified next.

Related:

Sources: