How to Use DTrace on FreeBSD for Live Kernel and Application Tracing

DTrace’s core value proposition is observing a running system — kernel and userspace both — without modifying, recompiling, or restarting anything being observed. On FreeBSD it’s a first-class, base-system-integrated facility, not a bolt-on tool, which matters because it means the providers instrumenting the kernel are genuinely part of the kernel rather than a separate module bolted on with more limited visibility.

Enabling DTrace

On most FreeBSD releases DTrace is available immediately, but confirm the necessary kernel modules are loaded:

kldload dtraceall

If DTrace was disabled at kernel-build time (uncommon on GENERIC kernels, more likely on a custom-built kernel with deliberately trimmed options), this will fail with a clear module-not-found error, at which point the fix is a kernel rebuild with options KDTRACE_HOOKS — but for the overwhelming majority of systems running the stock GENERIC kernel, this step just works.

Listing available providers

DTrace organizes its instrumentation points into providers — logical groupings of related probes. dtrace -l lists every currently available probe on the system, which on a typical system is tens of thousands of individual instrumentation points:

dtrace -l | wc -l
dtrace -l | grep syscall | head -20

You will essentially never write a script against the raw output of dtrace -l directly — it’s there to answer “does a probe exist for the thing I want to observe” while you’re constructing an actual script, not to be read start to finactualon.

A first real script: counting syscalls by process

The single most useful “first DTrace script” habit to build is syscall:::entry combined with an aggregation, which answers “what is every process on this system actually doing, syscall-wise, right now” without needing to know in advance which syscall matters:

dtrace -n 'syscall:::entry { @[execname] = count(); }'

Let this run for a few seconds and interrupt it (Ctrl-C) to print the aggregated counts — @[execname] = count() builds a table of process names to syscall counts, letting DTrace’s aggregation engine do the counting in-kernel rather than piping raw event data to userspace and counting there, which is both far more efficient and is exactly the pattern (aggregate in-kernel, only ship the summarized result to userspace) that makes DTrace usable on live production systems without the observation itself becoming the performance problem.

Finding which specific syscall a process is stuck in

When a specific process appears hung or is behaving unexpectedly, narrowing from “all syscalls, all processes” down to one process’s actual syscall sequence in real time is the natural next step:

dtrace -n 'syscall:::entry /execname == "myapp"/ { printf("%s\n", probefunc); }'

The /execname == "myapp"/ predicate restricts the probe to firing only for the process matching that name, and probefunc gives the specific syscall name — watching this scroll live for a hung process very often immediately reveals it’s blocked in a specific read() or connect() call, which redirects the investigation toward whatever resource that syscall is waiting on (a network connection, a file, a pipe) rather than continuing to treat “the process is hung” as an undifferentiated mystery.

Profiling where CPU time actually goes

DTrace’s profile provider fires at a fixed sampling interval regardless of what’s executing, which is the basis for statistical CPU profiling — sampling the currently-executing function across the whole system (or a specific process) at regular intervals builds up a picture of where time is actually spent, without the overhead of instrumenting every single function call:

dtrace -n 'profile-997 /pid == $target/ { @[ufunc(uregs[R_RIP])] = count(); }' -p <pid>

This samples the target process 997 times per second (a prime number is traditionally used to avoid accidentally sampling in lockstep with some other periodic system activity) and aggregates by which userspace function was executing at each sample. After letting it run and interrupting it, the aggregation printed shows which functions accounted for the largest share of samples — a direct, low-overhead answer to “where is this process actually spending its CPU time” without needing a separate profiling build or any code changes.

Tracing kernel function entry/return directly

The fbt (function boundary tracing) provider instruments entry and return of essentially any kernel function, which is useful for understanding kernel-level behavior that has no dedicated higher-level provider:

dtrace -n 'fbt::vfs_mount:entry { printf("mount attempt\n"); }'

This fires every time the kernel’s mount-handling function is entered, regardless of which filesystem or userspace command triggered it — useful for confirming, precisely, whether and when a specific kernel code path is actually being exercised, rather than inferring it indirectly from higher-level symptoms.

A practical script structure worth reusing

Most useful ad-hoc DTrace investigations follow the same shape: pick a provider/probe matching the event category you care about (syscalls, kernel functions, scheduler events), add a predicate to narrow it to the specific process/condition under investigation, and aggregate rather than print raw per-event output unless you specifically need per-event detail (printing every single event on a busy system floods the terminal and often drops probe firings under load, whereas an aggregation collapses volume in-kernel and never drops data from the events it does capture). Writing throwaway one-liners in exactly this pattern — provider, predicate, aggregation — covers the large majority of real diagnostic questions without needing to write and save a persistent .d script file at all, which is worth internalizing before reaching for DTrace’s full scripting capabilities on problems that a single well-aimed one-liner would answer just as well.