Linux Dynamic Debug: Enable pr_debug Call Sites Without Rebuilding

Linux dynamic debug lets an operator enable selected pr_debug() and dev_dbg() call sites at runtime when the kernel or module was built with dynamic-debug support. It is useful when a subsystem already contains diagnostic call sites but ordinary kernel logs do not include them. It does not inject new tracing into arbitrary code, and it does not replace tracepoints, ftrace, eBPF, or a rebuild when the needed diagnostic statement does not exist.

The key operational advantage is selection. Rather than enabling a noisy subsystem everywhere, query call sites by module, source file, function, line range, format string, or a declared class where supported. Start narrowly, collect the evidence needed, then disable the call sites again. Dynamic debug output is still kernel log output, so broad queries can flood the ring buffer, add overhead, or expose values developers did not intend to persist.

Confirm that the kernel supports dynamic debug

Check whether the control file is present:

test -r /proc/dynamic_debug/control \
	&& head -n 5 /proc/dynamic_debug/control

The kernel documentation identifies /proc/dynamic_debug/control as the interface when dynamic debug is enabled. The underlying call-site table is populated from code compiled with the relevant configuration. If the control file is absent, verify the running kernel configuration and the distribution’s kernel build; creating an empty file or mounting a filesystem will not add support to a kernel that lacks it.

Entries identify source location and call-site metadata. Inspect the table before enabling anything so you know which module, file, function, and format string actually match. A driver may be built into the kernel or loaded as a module, and a source filename may differ from the installed module’s user-facing name. Do not assume a selector based on a guessed module name will match.

Enable a narrow set of call sites

Write one query to the control file. For example, select one source file or one function and set the print flag:

printf '%s\n' 'file drivers/usb/core/hub.c +p' \
	> /proc/dynamic_debug/control
printf '%s\n' 'func usb_probe_interface +p' \
	> /proc/dynamic_debug/control

The +p flag enables printing for matching call sites. Quote the query so the shell does not expand wildcard patterns before the kernel receives them. Dynamic debug supports selectors for fields including module, file, function, line range, format string, and class when the code declares one. Use the syntax documented by the target kernel; paths and class names are case-sensitive inputs, and the call-site inventory is the best way to avoid a query that silently matches nothing.

A broad query such as enabling every call site in a busy driver can generate a large volume of messages. Prefer one function or a small line range, reproduce the issue once, then inspect the matching records with dmesg or the system’s kernel-log viewer. If the messages do not appear, confirm the query matched call sites, verify the facility is configured, and check the system’s log collection and rate-limiting behavior before broadening the filter.

Disable what you enabled

Disable a selector with -p:

printf '%s\n' 'file drivers/usb/core/hub.c -p' \
	> /proc/dynamic_debug/control

Dynamic debug is mutable kernel state. Record the exact selector used, the time window, and the original call-site flags when a test must be reproducible. Do not rely on reboot as the only cleanup method: a built-in query may be set on the boot command line, and a module-specific setting may be applied whenever that module loads. Remove or revise persistent configuration after the diagnostic run, then inspect the control table to confirm the expected flags.

If you are investigating a module that is not loaded yet, use the module’s supported dynamic-debug parameter or configure a query through the documented boot-time mechanism for that kernel. For a built-in subsystem, the kernel command line supports dyndbg="QUERY" syntax. Distribution bootloader tooling and quoting rules differ, so update the actual boot entry using the platform’s supported process and keep a known-good fallback entry. Do not copy shell redirection syntax into a bootloader command line.

Understand which messages can be enabled

Dynamic debug acts on registered debug call sites; it does not make all printk() messages conditional or turn arbitrary functions into probes. Kernel code commonly uses pr_debug() and device-scoped dev_dbg(). The kernel’s dynamic-debug documentation also covers class-based debugging for subsystems that declare classes. A dev_dbg() message can include device context, while a plain pr_debug() site may not identify the device instance without fields in its format string.

If a message was compiled out rather than registered, no selector can recover it. Inspect the matching kernel source and build configuration. For user-space logs, use that process’s logging facilities; writing to /proc/dynamic_debug/control affects kernel call sites, not printf() calls or application log levels.

Choose the instrumentation that fits the question. Dynamic debug is well suited to enabling existing branch-level diagnostics. Tracepoints provide structured events designed for tracing; function tracing observes entry/exit paths; eBPF can aggregate or filter events programmatically; and a debugger may be needed for state that no log site exposes. Combining tools can help, but collect a baseline first because tracing and verbose logging can change timing.

Avoid diagnostic logging hazards

Treat debug output as potentially sensitive operational data. Call sites may print addresses, identifiers, protocol values, paths, or device state. Review the selected format strings and restrict access to logs according to the system’s normal controls. Do not enable broad debug output on a shared production host without a retention, volume, and privacy plan.

The printk ring buffer is bounded. A burst of debug messages can overwrite earlier messages, compete with other kernel records, and increase CPU work. Avoid enabling high-frequency call sites during a severe incident unless the expected volume is understood. Prefer a controlled reproduction, a low-rate filter, or a trace mechanism that can aggregate events before output.

Plan how records will be correlated and retained before reproducing a failure. Capture the kernel release, boot identifier, monotonic timestamps when available, the exact query, and the start and stop times. On a system using systemd-journald, journalctl -k -b can inspect kernel messages from the current boot; on other systems, use the configured kernel-log reader. A journal that persists across reboot still needs enough storage and retention to keep the burst, while a ring buffer alone may be lost or overwritten. Export only the relevant interval through the approved logging path and redact sensitive values according to policy.

Dynamic debug is not a security boundary. A user who can change the control file already has privileged kernel-debugging capability, and the facility does not sanitize message content. Use the least broad selector that answers the diagnostic question, and stop collecting when the evidence is sufficient.

A repeatable diagnostic sequence

Record the kernel release, module state, and symptom time window.
Confirm the control interface exists and locate candidate call sites.
Select one file, function, or line range; enable only +p for that query.
Reproduce the issue for a bounded interval and preserve the relevant log records.
Disable the same selector, verify the flags, and check the next log window for continued output.
Remove temporary boot or module configuration and document what was learned.

When a query has no effect, inspect the actual table entry and the interface’s write response rather than immediately enabling every call site. When logs become excessive, disable the selector first and then narrow it. This keeps the diagnostic process reversible and makes the resulting evidence easier to correlate with the failure.

Dynamic debug is a focused switch for debug statements the kernel already compiled into a controllable call-site table. Verify support, query narrowly, account for log volume and data sensitivity, and clean up both runtime and persistent settings when the investigation ends.

Related:

Sources: