eBPF Explained: Safe, Programmable Observability in the Linux Kernel

eBPF is a Linux kernel execution and attachment framework used for tracing, networking, control groups, security hooks, scheduling extensions, and more. Userspace loads typed bytecode, the kernel verifier analyzes it under program-specific rules, and an interpreter or JIT executes accepted instructions when an attached hook fires.

The verifier reduces risk; it does not make arbitrary eBPF code harmless, mathematically proven, or suitable for untrusted tenants. Accepted programs execute in kernel context with the access allowed by their type, helpers, maps, credentials, and attachment point.

From packet filters to a typed kernel framework

Classic BPF provided a compact packet-filter language. Linux eBPF expanded registers, instructions, maps, helpers, program types, BTF typing, and attach mechanisms. “Extended Berkeley Packet Filter” remains common history, but kernel documentation generally calls the subsystem BPF/eBPF.

The framework is not one universal virtual machine permission set. A socket filter, tracepoint program, XDP program, cgroup hook, and BPF LSM program receive different context types and can call different helpers.

Loading is a privileged security decision

Modern kernels can separate BPF operations with CAP_BPF, performance-observation access with CAP_PERFMON, and network administration with CAP_NET_ADMIN; older kernels and some operations rely on CAP_SYS_ADMIN. The exact checks depend on program/map/attach type, kernel version, user namespace, and lockdown/LSM policy.

Inspect policy without changing it:

uname -r
cat /proc/sys/kernel/unprivileged_bpf_disabled
bpftool feature probe unprivileged

Many distributions disable unprivileged bpf() by default. Do not set the sysctl to 0 just to make a tutorial work, and do not grant broad capabilities to an entire observability agent without mapping its required operations.

The verifier performs abstract interpretation

Before loading, the verifier explores program states and tracks register types, scalar bounds, pointer provenance, stack initialization, control flow, helper contracts, and resource limits. It rejects accesses it cannot prove valid under its model. Bounded loops and function calls are supported subject to version and complexity limits.

This is not a mathematical proof of the entire kernel or intended program semantics. Verifier and JIT vulnerabilities have existed, helpers can have side effects, and a logically wrong but verifier-safe program can drop packets, deny access, or expose sensitive observations.

Program type defines context and authority

Every program is declared with a type and usually an expected attachment family. XDP programs receive an XDP context near the driver receive path. Tracepoints expose documented event layouts. kprobes attach to implementation details. LSM programs participate in security hooks when the kernel is configured for BPF LSM.

The type gates helpers and return semantics. Returning the wrong action from an enforcement hook can block traffic or operations at host scale. Test with a narrow target and documented fail-open/fail-closed behavior.

Prefer stable hooks over internal functions

Tracepoints and BTF-enabled fentry/fexit attachment are generally more maintainable than a kprobe on an internal symbol. Kernel function names, arguments, inlining, and prototypes change; a copied kprobe/do_sys_open example may simply be obsolete on a current kernel.

Inventory supported hooks and BTF rather than guessing:

test -r /sys/kernel/btf/vmlinux && echo BTF-present
bpftool btf list
bpftool prog help

These commands can require privilege and reveal system structure; run them only on authorized hosts.

Maps hold shared kernel-managed state

BPF maps store counters, policies, connection state, stacks, queues, ring buffers, or references depending on map type. Programs and userspace can access them under type-specific concurrency and lifetime rules. Per-CPU maps reduce contention but require aggregation; LRU maps evict; ring buffers stream events; map-in-map enables policy swaps.

Keys and values have fixed definitions known to the kernel. A map is not automatically private: pinning it in bpffs gives it a filesystem-visible lifetime, and permissions/namespace exposure matter.

bpftool map list
mount | grep ' type bpf '

Do not dump production maps casually; they can contain addresses, process metadata, network identities, or security policy.

Attachment links determine lifetime

A loaded program does nothing until attached, except program types invoked explicitly. Attachments may be represented by BPF links, perf events, netlink configuration, cgroup files, or subsystem-specific objects. Closing a loader can detach some legacy attachments, while pinned links/programs/maps can survive it.

Audit program, map, link, and network attachment state together:

bpftool prog list
bpftool link list
bpftool net list

An empty loader process list does not prove no BPF remains attached.

JIT and interpreter are implementation choices

Supported architectures can JIT bytecode to native instructions; others or policy configurations may interpret it. JIT hardening, symbol exposure, memory limits, and always-on behavior are controlled by kernel build/runtime policy.

Do not disable hardening to chase a benchmark without a threat-model review. Performance depends on hook frequency, program instructions, helper cost, map contention, cache behavior, event transport, and userspace consumption—not merely “BPF runs in the kernel.”

BTF and CO-RE improve portability, not universality

BPF Type Format describes kernel and program types. Compile Once – Run Everywhere uses BTF relocations plus libbpf so field offsets and type differences can be adapted when the target supplies compatible information. It reduces the need for target kernel headers/runtime compilation.

CO-RE cannot make a missing hook, helper, map type, or semantic behavior appear. Feature-probe the target, define minimum kernels, and implement graceful fallback. Preserve compiler, libbpf, object, BTF, and kernel provenance for reproducibility.

Use libbpf and maintained skeletons

Raw bpf() syscalls, ring memory barriers, object loading, relocations, attachment cleanup, and feature compatibility are easy to get wrong. The kernel community’s libbpf and libbpf-bootstrap demonstrate current CO-RE patterns, generated skeletons, ring buffers, and cleanup.

Pin dependencies and verify source provenance. Do not compile and run an arbitrary one-line tracing script as root on production; high-level tools still generate privileged programs and can collect secrets.

Observability can change the system

Tracing every syscall or file open can generate enormous event volume, consume CPU, fill maps/ring buffers, and perturb the latency being measured. bpf_printk() is intended for debugging and trace output, not production telemetry. Prefer in-kernel aggregation, sampling, bounded cardinality, and loss counters.

Treat command lines, filenames, socket addresses, stack traces, and process identities as sensitive. Apply minimization and retention policy before events leave the host.

Networking programs are active data-plane code

XDP can pass, drop, redirect, or transmit packets before much of the normal network stack. TC and cgroup programs can classify or modify traffic. A verifier-accepted logic bug can blackhole a host or bypass an intended userspace policy path.

Deploy to one interface/cgroup, retain an out-of-band management path, and make rollback independent of the affected network. Validate MTU, fragments, IPv4/IPv6, checksums, offloads, tunnels, and failure modes.

Security programs need fail-mode design

BPF LSM and cgroup hooks can enforce decisions, but the loader, pinned policy maps, update authority, and detach behavior become part of the trusted computing base. Decide what happens if the agent exits, map fills, userspace falls behind, kernel upgrades, or policy update is partial.

Capabilities are not a full containment strategy. Combine a minimal bounding set with seccomp, LSM rules, read-only filesystems, protected bpffs pins, service isolation, signed packages, and audited update paths.

Diagnose verifier rejection without weakening policy

Capture the kernel version/config, bpftool/libbpf/compiler versions, program type, attach target, BTF presence, required helpers/maps, privilege sets, lockdown state, and complete verifier log. A verifier error often means pointer state or feature mismatch, not “permission denied.”

Never solve rejection by disabling unprivileged-BPF restrictions, lockdown, or signature policy globally. Reproduce in a disposable matching kernel, minimize the program, and consult current kernel documentation.

Build a forensic acceptance test

Record hashes of loader/object/config, all capabilities, program/map/link IDs and tags, pins, attachments, map limits, expected event rate/loss, data fields, and exact detach command. Load against a narrow disposable target, trigger known events, test malformed/high-rate input, then detach and prove IDs/pins disappear.

The deployment is safe enough to consider when authority is minimal, target scope is explicit, data collection is bounded, behavior survives a version probe, rollback works without the monitored path, and no one calls verifier acceptance a sandbox guarantee.

Related:

Sources: