Diagnosing a FreeBSD Kernel Panic from a Crash Dump

A FreeBSD panic becomes diagnosable only if the host preserves evidence across reboot. The kernel writes a dump to a configured raw dump device; early in the next boot, savecore(8) extracts it into /var/crash; crashinfo(8) produces a first summary; and kgdb combines the dump with symbols from the exact crashed kernel. A stack containing a driver’s name is a lead, not proof that the driver caused the fault.

Verify dump capture before the next incident

Inspect runtime state as well as rc.conf:

sysrc dumpdev
sysrc dumpdir
dumpon -l
swapinfo -h
df -h /var/crash
ls -ld /var/crash

dumpdev="AUTO" asks the rc system to select an appropriate swap device; an explicit provider is clearer on hosts with several swap devices. dumpon -l reports the active kernel configuration and catches cases where rc.conf looks correct but activation failed. The dump device must be large enough for the chosen dump type, and /var/crash needs enough free space for extraction.

FreeBSD defaults to minidumps on supported architectures, which preserve kernel-relevant pages and are often much smaller than full physical memory. Full dumps and textdumps have different purposes and capacity requirements. Do not size storage from one old rule of thumb without checking the running release and dumpon(8).

Crash dumps can contain passwords, keys, application data, and user content from memory. Keep /var/crash root-only, restrict backups and support uploads, and consider the public-key encryption facility documented by dumpon(8) when the threat model requires it. Verify that the corresponding private key recovery process actually works; encryption without recoverable keys converts evidence into unusable data.

Do not test by deliberately panicking a production host. The Developers’ Handbook documents debug.kdb.panic=1 as a dump test, but it immediately crashes the kernel. Use it only in an approved maintenance window on a disposable or redundant system, ideally from single-user mode with filesystems protected as the guide describes.

Preserve the first-boot evidence

After reboot, inspect the extraction service and directory:

service savecore status
ls -lah /var/crash
readlink /var/crash/vmcore.last
sed -n '1,220p' /var/crash/core.txt.last

Exact symlink names and files depend on release and configuration. savecore numbers dumps to avoid overwriting an existing vmcore.N and creates vmcore.last; crashinfo normally writes core.txt.N and a link to the latest summary. If there is no dump, inspect boot messages and /var/log/messages for savecore, dump-device, capacity, or header errors before rebooting repeatedly.

Copy the dump, summary, kernel build identity, and logs to protected analysis storage before rotating or deleting anything. Record:

freebsd-version -kru
uname -a
sysctl kern.bootfile
sha256 /boot/kernel/kernel

The rebooted kernel may not be the crashed one. An administrator might boot kernel.old, activate another ZFS boot environment, or complete an update during recovery. Hashing only the current /boot/kernel/kernel does not prove it matches the dump; obtain the kernel and debug symbols from the build or boot environment that produced the panic.

Read crashinfo before opening a debugger

crashinfo(8) collects the panic string, stack trace, loaded modules, process and thread state, locks, memory information, and other commands useful for triage. Start there because it is reproducible and easy to attach to a bug report after checking it for secrets.

Identify the first concrete failure, not merely the final panic() frame. Common classes include page faults, assertion failures, lock-order reversals promoted to panic, watchdog timeouts, double faults, and explicit subsystem consistency checks. Record the panic text exactly. Search the trace for the transition from generic trap/panic code into subsystem code.

Review messages immediately before the panic for I/O errors, device resets, allocation failures, watchdog warnings, ZFS reports, or hotplug events. Wall-clock logs may not contain the final buffered lines, while the in-memory message buffer in the dump can. Treat both as partial evidence.

Match kgdb to the crashed kernel

Useful source lines and local variables require debug symbols from the exact kernel build. A GENERIC debug distribution may provide kernel.debug; a source-built kernel normally leaves it in that build’s object directory. Kernel modules also need matching symbols. Do not analyze a 15.1 dump with a later patched kernel simply because both say “15.1.”

The documented form is:

kgdb /path/to/matching/kernel.debug /var/crash/vmcore.last

Inside kgdb, begin conservatively:

(kgdb) bt
(kgdb) bt full
(kgdb) info threads
(kgdb) frame 6
(kgdb) list

Choose the relevant frame shown by the actual backtrace; frame 6 is only illustrative. Optimized kernels may report variables as <optimized out>, and corrupted stacks can produce misleading frames. A symbol mismatch commonly appears as implausible function names, missing module symbols, or addresses that do not map cleanly.

The faulting thread is not always the causal thread. An assertion may detect corruption created earlier, and a page fault inside a driver may result from bad memory supplied by another subsystem. For deadlocks or multi-CPU interactions, collect all thread backtraces and lock state from crashinfo/kgdb. For hardware suspicion, correlate machine-check, ECC, PCIe AER, storage, and management-controller logs.

Turn a backtrace into a testable hypothesis

Separate observation from conclusion:

Observation: exact panic string, fault address, current thread, first non-generic frames, loaded module versions.
Correlation: workload, device event, update, configuration change, or repeatable trigger immediately preceding it.
Hypothesis: a specific invariant or pointer appears violated in one subsystem.
Test: reproduce on supported patched software, remove one suspected variable, or add targeted diagnostics.

Do not unload a storage or network driver on a production system merely because its function appears in the stack. Check the matching release’s errata, Security Advisories, UPDATING, and FreeBSD bug database. A fix may already exist in a supported point release or stable branch. Conversely, upgrading everything before preserving the dump can erase the exact binary provenance developers need.

For a useful report, include FreeBSD version and source revision, architecture, hardware and firmware, kernel configuration, loaded modules, reproduction steps, panic text, sanitized crashinfo, and a symbolized backtrace. State whether the issue repeats and whether memory or hardware diagnostics found errors. Do not publish the raw vmcore: it is large, sensitive, and usually unnecessary unless a trusted developer explicitly requests a secure transfer.

Repair the capture path as part of resolution

If the incident produced no dump, that is a second defect. Correct dump-device activation, space, permissions, and symbol retention before closing the case. Keep debug artifacts for every deployed custom kernel, tied to its source revision and configuration. Monitor /var/crash capacity and alert when savecore fails.

The highest-value action is preparation: an active dump device, protected extraction directory, matching symbols, and a practiced collection procedure. With those in place, a panic changes from an unexplained reboot into a timestamped, symbolized state that can support a real root-cause investigation.

Related:

Sources: