Tracing System Calls and Kernel Events with bpftrace

bpftrace is a high-level tracing language built on eBPF that lets you attach lightweight probes to kernel functions, tracepoints, and syscalls, and run small snippets of logic every time they fire — aggregating data, printing events, or computing histograms — all without recompiling anything or loading a kernel module. It’s the practical, day-to-day entry point into eBPF-based observability, distinct from writing raw eBPF programs by hand.

Why reach for this instead of strace

strace intercepts syscalls by stopping the traced process at every syscall boundary via ptrace, which is straightforward but carries meaningful overhead — often significant enough to change the timing behavior of whatever you’re debugging, sometimes enough to mask race conditions or performance issues you’re trying to observe. bpftrace probes run inside the kernel itself, attached to the specific event you care about, with per-event overhead low enough to trace busy production systems without materially disturbing their behavior — the right tool when strace’s overhead itself would distort the very thing you’re investigating, or when you need to trace kernel-internal behavior that never surfaces as a syscall at all.

The one-liner: tracing a specific syscall system-wide

bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args.filename)); }'

This attaches to the openat syscall’s entry tracepoint and prints the calling process’s name and the filename being opened, for every single process on the system making that call — a live, system-wide view of file-open activity, useful for questions like “what process keeps opening this specific config file” without needing to know the PID in advance or attach strace to a specific process.

Counting events instead of printing every one

For high-frequency events, printing every occurrence floods the terminal uselessly. Aggregating counts is usually more useful:

bpftrace -e 'tracepoint:syscalls:sys_enter_read { @reads[comm] = count(); }'

The @reads[comm] = count() pattern builds a running, per-process-name count of read() syscalls, printed automatically (sorted, by default) when the script is stopped with Ctrl-C — directly answering “which processes are making the most read syscalls” without manually tallying individual trace lines yourself.

Histograms: understanding latency distribution, not just averages

One of bpftrace’s most genuinely useful patterns is timing an operation’s distribution, since an average can hide a bimodal pattern (mostly-fast with an occasional severe outlier) that matters far more in practice than the mean:

bpftrace -e '
kprobe:vfs_read { @start[tid] = nsecs; }
kretprobe:vfs_read /@start[tid]/ {
    @latency_ns = hist(nsecs - @start[tid]);
    delete(@start[tid]);
}'

This attaches to both the entry and return of the kernel’s vfs_read function, records a timestamp on entry, and on return computes the elapsed time and buckets it into a power-of-two histogram — printed as a readable distribution on exit, immediately showing whether read latency is consistently fast with a long tail of outliers, or something else entirely, in a way a single averaged number never could.

Tracing a specific process instead of system-wide

Adding a filter predicate scopes a probe to just the process you care about, cutting noise from everything else running on the system:

bpftrace -e 'tracepoint:syscalls:sys_enter_write /pid == 4821/ { printf("%s wrote %d bytes\n", comm, args.count); }'

The /pid == 4821/ predicate — bpftrace’s filter syntax — restricts the probe to firing only for that specific PID, which matters enormously on a busy system where the unfiltered version would otherwise be dominated by unrelated processes’ activity.

Using one of the bundled example scripts instead of writing your own

bpftrace ships (via the bpftrace-tools or equivalent package depending on distribution) with a library of pre-written scripts for extremely common questions, which is worth checking before writing a one-liner from scratch:

/usr/share/bpftrace/tools/opensnoop.bt
/usr/share/bpftrace/tools/tcpconnect.bt
/usr/share/bpftrace/tools/biolatency.bt

opensnoop.bt traces file opens system-wide with useful formatting already built in; tcpconnect.bt traces outbound TCP connection attempts; biolatency.bt produces exactly the kind of block-IO latency histogram the hand-written example above builds manually — these bundled scripts cover a large fraction of common tracing needs without requiring you to write bpftrace syntax at all.

What requires elevated privileges, and why

Loading eBPF programs into the kernel requires root (or, on modern kernels, specific capabilities like CAP_BPF plus CAP_PERFMON rather than full root, if configured) — this is a deliberate, meaningful restriction, since an eBPF program attached to a kernel-internal function has visibility into essentially everything happening on the system. Running bpftrace scripts as an unprivileged user will fail outright, which is expected behavior rather than a bug to work around.

The general shape of a bpftrace investigation

The recurring pattern across genuinely useful bpftrace usage is: start from a specific, concrete question (“which process is opening this file,” “what’s the actual read-latency distribution,” “which connections is this process making”), find or write the smallest probe that directly answers it, and only reach for a bundled tool or a custom multi-line script once a plain one-liner turns out to be insufficient — the tracing language is powerful enough to build arbitrarily elaborate scripts, but the majority of real diagnostic value comes from short, targeted probes aimed at a specific question rather than broad, exploratory tracing.