FreeBSD User Process Core Dumps: Capture, Limits, and Analysis

A user-process core file is a snapshot written when a process terminates because of a signal whose default action includes a core dump. Developers and operators can inspect that memory image later with a debugger. Capturing one reliably requires more than finding a file named core: the process must be permitted to dump, the destination must be writable and large enough, the filesystem must have capacity, and the exact executable and libraries must be preserved for meaningful analysis.

This article covers userland process cores, not kernel crash dumps. A user core represents one process address space. A kernel vmcore represents system state after a kernel panic and is handled through a different path involving dump devices, savecore, and crash analysis. The two artifacts differ in size, permissions, trigger, and analysis tools. Do not use a process core procedure to investigate a system reboot after panic.

Understand whether a process can dump

The process resource limit RLIMIT_CORE sets the maximum size of a core file. If a dump would exceed the limit, FreeBSD does not create the file. The limit can be inherited from the shell, login class, service manager, or the process itself. A shell’s coredumpsize limit is one way to control it, but changing an interactive shell does not retroactively alter a running daemon.

Check both the process launch environment and the live process. Before starting a test application, inspect the current shell limit:

ulimit -c
ulimit -a

For a running process, procstat rlimit PID reports resource limits as observed by the kernel. This is more relevant than the shell that an operator happens to have open. If a service manager starts the program, inspect the service’s own environment and limits rather than assuming a login shell’s settings apply.

Core generation also depends on the signal and process credentials. Not every termination creates a core. A process that changes user or group credentials is restricted by default from producing one; FreeBSD exposes a tunable to change that behavior, but enabling it should not be a casual troubleshooting step. Preserve this as a distinct policy decision rather than interpreting the absence of a file as proof that a crash did not occur.

Choose a destination and naming pattern

By default, FreeBSD names a process core after the program and writes it in the process’s current working directory if the process has permission there. This is convenient for a developer running a local test and unreliable for a daemon that starts in /, a read-only path, or a directory cleaned automatically. The kern.corefile sysctl controls the filename pattern. It can be an absolute path or a relative path resolved against the crashing process’s working directory.

The manual supports format fields including hostname, process name, process ID, signal, user ID, and a bounded index. Naming by PID and process name helps distinguish concurrent crashes; adding a user ID helps organize per-account artifacts. Avoid a pattern that causes unrelated instances to overwrite the same pathname.

For a service with a dedicated numeric UID, create an administrator-owned destination in advance, assign the service only the directory access required to write its own core, then configure a pattern whose directories actually exist. The manual’s per-user example assumes the corresponding directory is present and writable. Setting a path containing a UID placeholder does not create that directory automatically.

For an isolated lab, the runtime setting can be inspected and changed with sysctl:

sysctl kern.corefile
sysctl kern.corefile=/var/coredumps/%U/%N-%P.core

This pattern is only usable when /var/coredumps and the matching user subdirectory exist with appropriate permissions. Test it with a nonproduction program in a disposable environment, and verify the produced filename and owner. A service that cannot write into the selected directory will not produce the expected artifact.

To preserve the setting through reboot, add the exact reviewed assignment to the normal sysctl configuration mechanism for the installed release. Keep the required directory creation and ownership setup in a separate, ordered service or provisioning step. Do not assume that sysctl configuration creates the directory tree. A boot-time race can make the setting correct while the first early-starting process still has no usable destination.

Capacity, quotas, and duplicate dumps

Core files can approach the process’s address-space size and may take a long time to write. Plan for peak concurrent failures, not just one average dump. Check the destination filesystem’s free space, user or filesystem quotas, and the process’s core limit. A full filesystem can affect unrelated services if the core directory shares a volume with logs or application data.

FreeBSD supports an index substitution that increments from zero up to the debug.ncores limit, which can limit how many core files are retained per process naming pattern. Confirm the precise running-release behavior before relying on it as retention. It does not replace a disk-capacity budget, periodic cleanup, or an archival policy. Test concurrent crashes in staging to prove that files do not collide and that the service remains available.

Core compression is available only when the kernel has the appropriate compression I/O option and runtime controls are configured. Do not assume that a large dump will compress automatically on every kernel. If compression is enabled, check the suffix and tooling expected by the analysis pipeline. Compression can save storage but consumes CPU and may increase time before a diagnostic artifact is ready.

Preserve the executable and its context

A core is most useful when paired with the exact executable that produced it, matching shared libraries, build identifiers, and debug symbols. A later package upgrade may replace the binary while retaining the old core. Record the binary’s checksum, package version, build ID if available, operating-system release, architecture, loaded libraries, and service command line at crash time.

If the program came from ports, the base system may not include the GDB executable. Install or provision the appropriate debugger and debug symbols from the same build pipeline. A typical GDB invocation is:

gdb /usr/local/bin/example-service /var/coredumps/1001/example-service-1234.core

Replace both paths with the saved executable and exact core. Start with the backtrace and thread inventory, then inspect the faulting frame and relevant variables. For a multithreaded program, a backtrace for every thread often shows whether the crash occurred in the main event path, a worker, or a library callback. Optimized binaries may have incomplete local variables even when the correct symbol file is present.

A debugger warning about an executable or shared library mismatch is meaningful. Do not force conclusions from a symbolic backtrace produced using the current binary if the core came from an earlier release. Recover the matching build artifact or symbol package. Without symbols, raw addresses may still help correlate with a map file, but they are not a trustworthy source-level diagnosis.

Handle dumps as sensitive operational artifacts

A process core can contain in-memory data from active requests, configuration, cached secrets, session state, or customer information. Limit who can read the destination, avoid attaching a raw core to public bug reports, and follow the site’s retention policy. This is an operational data-handling requirement even when the investigation itself is routine.

Core paths may include user-supplied process names or other values depending on the naming pattern. Make sure the destination cannot be filled through uncontrolled process churn. Monitor free space and count core artifacts, keep logs about their creation, and clean them only after an authorized retention period. If the process repeatedly crashes, rate-limit restart behavior through the service manager and preserve representative samples instead of generating unlimited dumps.

Why a core may be missing or unusable

No core file can result from a zero core-size limit, insufficient destination permissions, absent UID subdirectory, full filesystem, quota, a signal that does not dump, a credential transition, or a process-specific dump policy. Also verify that the path is not on a filesystem mounted with restrictions incompatible with the service. Capture the process’s actual resource limits and sysctl kern.corefile value before changing policy.

A tiny or incomplete artifact may indicate a size limit or write failure. A dump that appears to be a kernel vmcore should be classified separately. A file that exists but cannot be opened may be compressed, truncated, from another architecture, or paired with the wrong executable. Use file, file size, debugger diagnostics, and system logs to determine which layer failed.

The absence of a core does not prove that the process never crashed; service supervision may restart it, logs may record a termination signal, or the filesystem could have rejected the dump. Likewise, the presence of a file named after the program does not prove that it is complete or from the most recent incident. Correlate timestamps, PIDs, service logs, and build identifiers.

Validate the capture path safely

Never force a production daemon to crash merely to validate core configuration. Use a test process in a disposable VM or staging host with the same service manager, account, resource limits, mount layout, and sysctl settings. Generate a known test termination only with an approved diagnostic binary, confirm the dump path and permissions, open it with the matching executable, then remove it according to policy.

Acceptance evidence includes the live process limit, active kern.corefile pattern, existence and ownership of destination directories, available capacity, observed core filename and size, matching executable hash, successful debugger load, and cleanup behavior. Re-test after package or kernel upgrades because process limits and compression support can change with the deployment path.

The useful artifact is not merely a file on disk. It is a reproducible pair of process memory and exact runtime context, retained long enough for analysis but bounded so a failure loop cannot consume the host. Separating capture, storage, symbol matching, and retention makes process-core diagnosis dependable and keeps it distinct from kernel crash analysis.

Related:

Sources: