FreeBSD ktrace and kdump: Reading Process Behavior from Kernel Records

When a FreeBSD program fails before it produces a useful log, the kernel’s view of its system calls can answer questions that application output cannot. ktrace enables selected trace points for a process, process group, or command; kdump decodes the resulting binary records into a readable sequence. Together they expose behavior at the user/kernel boundary without requiring a debugger or a custom instrumented build.

They are diagnostic instruments, not passive audit systems. A trace can grow rapidly, may contain command arguments or environment values, and can record activity from descendants when inheritance is enabled. Decide what question the trace must answer, restrict its scope, and learn how to stop it before collecting production data.

Trace points are a filter, not a query language

The current FreeBSD 15.1 ktrace(1) manual documents separate one-letter trace classes for system calls (c), page faults (f), I/O (i), pathname translation (n), capability-check failures (p), signals (s), selected structures (t), userland utrace(2) records (u), context switches (w), sysctl(3) requests (y), execve(2) arguments (a), execve(2) environment variables (e), and extended kernel errors (x). Those classes are not a query language: they select broad record families, not individual paths or system calls.

The + shorthand means the manual’s default set, not every available class. That set includes a c e i n s t u x and y; it omits, for example, page faults (f), capability failures (p), and context switches (w). For the common question “which calls and path lookups did this process make?”, -t cn is narrower than +. Add classes only when the hypothesis needs them.

A bounded collection workflow

Trace a single command into a dedicated file, then decode it with kdump:

trace_file=$(mktemp /tmp/ktrace.XXXXXX)
chmod 600 "$trace_file"

ktrace -f "$trace_file" -t cn -- /usr/local/bin/example --check
status=$?

kdump -f "$trace_file"
printf 'command exit status: %s\ntrace: %s\n' "$status" "$trace_file"

mktemp creates a unique file; the explicit mode narrows access to the owner. Review the decoded output before attaching it to a ticket. A process may pass credentials, tokens, filenames, or user data in system-call arguments. Treat the trace file like sensitive diagnostic material and remove it under your retention policy after the investigation.

The trace is binary and may contain data that is not obvious from the command line. The i class records I/O activity, and the a and e classes can expose arguments and environment values passed through execve(2). A trace may therefore contain secrets even when the command itself looks harmless. Avoid + for first-pass diagnosis, restrict file permissions, store the file outside shared temporary locations when policy requires it, and redact a copy before sharing. Keep an unchanged original only when the incident process requires evidence preservation.

For a live process, capture the selector and process identity before enabling tracing:

pgrep -x example
pid=12345 # Replace with the verified PID from the listing above.
ps -p "$pid" -o pid=,ppid=,user=,command=
ktrace -f "$trace_file" -t cn -p "$pid"

This is an operator workflow, not a race-free PID handle. The process can exit between inspection and the ktrace call, and a later process can reuse a PID. Re-check the command identity when a trace appears to contradict the expected service. For a short-lived command, tracing the command directly is usually more reliable than finding its PID after launch.

For a running service, -p pid selects one process. -g pgid selects a process group. The manual describes -d as applying the operation to current children of the selected process or group, while -i passes trace flags to future children. Use both only when current helpers and newly launched descendants are part of the question. This widens collection and may include workers unrelated to the failing request. Record the selected PID or group, parent process, command line, classes, and start time so the scope can be reconstructed later.

Decode chronology, not just individual calls

kdump renders records with process identity, event type, and call-specific data. Read the trace as a timeline. A failed open may be followed by a successful lookup of an alternate path; a SIGCHLD may explain why a parent is waiting; an execve record can show that a wrapper selected a different binary than expected. A single ENOENT is not proof that a file is absent from the whole system: the process may have tried a fallback path, run in a different root, or lacked permission to traverse a directory.

Use other tools to test hypotheses suggested by the trace. procstat can inspect process state and descriptors, while DTrace can aggregate behavior or instrument broader system activity. These tools answer different questions: ktrace gives a process-oriented record; it does not by itself explain scheduler latency, storage-device health, or every kernel path.

Stop tracing and contain the operational cost

The manual warns that trace output can become enormous quickly. The configured trace points remain active until the process exits or they are cleared. For a known process, clear its trace points narrowly:

ktrace -c -f "$trace_file" -p "$pid"

For a command launched only for a bounded reproduction, let it exit and confirm that no child inherited the trace flags. ktrace -C disables tracing on all user-owned processes, and when invoked by root it can disable tracing system-wide. That is a broad recovery control, not a routine cleanup command for a shared host. A broad clear can disrupt another operator’s investigation.

The utility also requires a kernel built with the KTRACE option. Check the installed kernel and the exact release manual before assuming the facility is available. If a command reports that tracing is unavailable, do not confuse that with a target-process failure.

Read records as a timeline, with limits

kdump can display elapsed or relative timestamps, filter output by PID, and suppress I/O payloads. For example, kdump -E -f "$trace_file" adds elapsed time from the start of the trace; kdump -p "$pid" narrows display to one process or thread identifier; and kdump -s suppresses I/O data. These options help make a large trace reviewable, but they do not make it a complete system-wide chronology. Records from concurrently traced processes may interleave, timestamps do not explain scheduler causality, and calls can be made by helper processes with different PIDs.

Read syscall entry and return records together. An open returning ENOENT can be followed by a fallback path that succeeds. A failed connect may reflect routing, a listener, or policy outside the process’s filesystem view. A signal record says that signal processing was traced; it does not by itself establish which component initiated the underlying event. Pair the trace with the service log, process state, filesystem permissions, network state, and the relevant configuration. Do not turn a correlation in the dump into a causal conclusion without another test.

An evidence checklist

Before collecting, record the host release, executable path, PID or process group, trace classes, start and stop times, and the exact reproduction command. Afterward, verify that the process completed or that tracing was cleared, that the trace is readable with kdump, and that the output answers the original question. Record the exact ktrace and kdump commands, selected classes, trace file ownership and mode, start/stop times, host release, and any redaction. Preserve an unmodified restricted copy only when the incident process requires it; make any redacted copy separately so its edits are auditable.

ktrace is most useful when the collection is narrow and repeatable. Its records can show what a process asked the kernel to do and what response it received. They do not automatically establish why the kernel made that decision, so pair the trace with the relevant configuration, filesystem state, permissions, and application context.

Related:

Sources: