FreeBSD netdump: Capture Panics on a Remote Host
Configure FreeBSD 15.1 netdump and its receiver, verify kernel and network prerequisites, protect captured cores, and test the panic path safely.
netdump(4) sends a FreeBSD kernel dump to a remote host after the client kernel panics. It is a recovery path for machines where local dump storage is unavailable, unreliable, or part of the failure domain, not a general-purpose backup protocol and not a substitute for checking that a dump can be analyzed. The local-dump workflow in the related panic guide configures a dump device and extracts evidence after reboot; this article instead covers the network client, netdumpd(8) receiver, and the trust boundary between them.
The protocol is UDP-based and the receiver is intentionally small, but that simplicity has operational consequences: the receiver does not authenticate clients and can be made to write arbitrary amounts of data to a mounted filesystem. Put it on a trusted, controlled network, budget receiver storage, and treat every dump as sensitive memory evidence.
Confirm kernel, driver, and address prerequisites
The FreeBSD 15.1 netdump(4) manual lists INET, DEBUGNET, and NETDUMP as kernel options for client support. It also lists the network drivers that support netdump; do not infer support from the fact that an interface works during normal operation. Confirm the exact NIC driver and release before planning a remote-dump path. The documented implementation supports IPv4 and can be used only after the kernel has panicked.
Choose a stable client address, server address, and route before configuring the client. dumpon(8) requires the selected interface to be up when netdump is configured. -c supplies the client’s local IP, -s the server IP, and -g an optional first-hop gateway. Without -g, FreeBSD uses a configured default route, or assumes the server is on the same link if there is no default route. The configured gateway is not automatically updated if routing topology changes.
Prepare a constrained receiver
netdumpd is supplied by the FreeBSD ports collection, not the base system. Create a dedicated directory on a filesystem with monitored capacity and a retention plan. The manual’s unprivileged foreground example is useful for a lab:
mkdir -p captures
cd captures
netdumpd -D -d . -P ./netdumpd.pid-D keeps the process in debugging mode and prints to the console. For a production receiver, follow the installed port’s service integration rather than leaving a diagnostic foreground process as an unmanaged daemon. The receiver’s documented default directory is /var/crash; -d selects another dump directory. Client-selected relative paths are resolved below dumpdir. The -p path option supplies a relative default destination for clients that omit a path. The manual says an invalid client path falls back to the “default directory,” but does not clearly distinguish dumpdir from the -p destination in that sentence. Test and document the exact installed port’s fallback behavior before relying on it as a retention or tenant-isolation boundary.
Restrict inbound client traffic to the intended hosts or isolated network. Netdump clients send the initial HERALD to server UDP port 20023; server messages are addressed to client port 20024. The server uses a random source port for its HERALD acknowledgement, and the client sends subsequent messages to that port, so account for this stateful return path in firewall rules. Verify the actual packet path with the network policy in use. A successful ping tests neither the documented UDP exchange nor the panicked-kernel driver’s ability to transmit.
Configure and inspect the client
Use the live dumpon(8) command first, with addresses and interface names replaced by values verified on the target:
ifconfig igb1
route -n get 192.0.2.20
dumpon -i 0 -c 192.0.2.31 -s 192.0.2.20 -g 192.0.2.1 igb1
dumpon -lThe documentation-only addresses above are from the IPv4 TEST-NET ranges; they are placeholders, not a working configuration. -i 0 places netdump first in the prioritized dump-device list. FreeBSD 13.0 and later can configure fallback dump devices, so a local dump can remain a later option when policy and storage permit. Verify the list with dumpon -l rather than assuming a command appended or replaced a prior setting.
For a fixed client address, dumpon_flags in rc.conf can persist netdump options. Boot-time dumpon runs early, typically before networking is configured; the manual warns that this is unsuitable when the client address is dynamic. In that case, configure netdump after DHCP binds, following the dhclient-exit-hooks example in the installed dumpon(8) manual. Do not put a guessed address or an interface name from another host into a production boot file.
Test capture without risking a service host
First confirm the receiver can write in its dump directory, the firewall permits the client/server exchange, and the selected driver is supported. Then use a disposable, DDB-enabled VM or a maintenance system with console access and a recoverable snapshot. FreeBSD’s dumpon(8) manual demonstrates a deliberately induced panic through sysctl debug.kdb.panic=1; for its interactive DDB test sequence, type db> dump and then db> reset to write the core and reboot. A test panic immediately interrupts the operating system, so never run it on a production system to “see if netdump works.”
After the test, verify that the receiver produced the expected information and core files, that sizes and timestamps are plausible, and that kgdb can read the dump with the matching kernel image and symbols. A configured device list or a normal-network ping is not proof of end-to-end panic capture. Preserve the client kernel build, driver, network path, dumpon -l output, receiver version, and test result with the artifact.
Protect the evidence after receipt
Kernel dumps can contain credentials, tokens, process memory, and user data. Restrict access to the receiver directory, encrypt backups, define retention, and make capacity alerts actionable before enabling multiple clients. FreeBSD supports encrypted kernel dumps through dumpon -k when the running kernel was built with EKCD; however, the manual explicitly cautions that the current encrypted dump scheme does not provide integrity or authentication. Encryption is a confidentiality control, not proof of sender identity or an intact artifact.
Related:
Sources: