FreeBSD Interrupt Diagnostics: Read vmstat Counters Before Pinning IRQs

Interrupts connect device events to kernel work. A storage controller, network adapter, timer, or other device signals that work needs attention; the kernel’s interrupt subsystem dispatches handlers and associated deferred processing. A high interrupt count can be normal for a busy device, while a low count can accompany a stalled or detached device. Counts alone do not diagnose a fault, identify a CPU bottleneck, or establish that interrupt affinity is wrong.

FreeBSD’s vmstat(8) provides a concise per-device interrupt report. It reports counts since system startup, so an operator must distinguish cumulative totals from changes over a time interval. cpuset(1) can query and, on supported hardware and configurations, modify an interrupt’s processor set. Treat affinity changes as an experiment with a narrow hypothesis, not a generic performance fix.

Record a baseline and identify the device

Start with host and driver context:

freebsd-version -kru
uname -a
devinfo -rv
dmesg -a | tail -100

Capture the relevant hardware path and driver messages before comparing counters. A label in the interrupt table may identify a device or handler, but it may not be a stable device identity across reboots, driver changes, or hardware enumeration. Record the device name, driver, interrupt identifier when present, and corresponding devinfo node.

Take a baseline with:

vmstat -i

The -i mode reports the number of interrupts taken by each device since system startup. vmstat -i -a includes entries that have never fired, which can help distinguish a registered-but-idle handler from one omitted in the default display. The default report can include totals and a rate column; do not infer that the total is a rate. The total accumulates over uptime, and reboot resets it.

To calculate a short-window change without assuming a sampling mode, capture two reports:

vmstat -i > /var/tmp/interrupts.before
sleep 10
vmstat -i > /var/tmp/interrupts.after
diff -u /var/tmp/interrupts.before /var/tmp/interrupts.after

Subtract the first total from the second for the same named handler. The interval is the actual elapsed time between command executions, not an exact scheduler measurement. Repeat the observation during a quiet baseline and the suspected workload. Do not compare totals from hosts with different uptime as if they were equivalent.

Interpret rows without overclaiming

The table reports device-associated interrupt accounting; it is not a per-process CPU profile and does not show all work performed after the handler runs. Drivers may defer processing to kernel threads, taskqueues, or network input paths. A NIC with a high interrupt count can reflect packet rate, receive queue configuration, traffic mix, or driver behavior. It is not automatically evidence of a storm.

Check the rest of the system at the same time:

vmstat -P
systat -vmstat 1
top -SH
netstat -i

Use the installed manuals for exact display behavior. Per-CPU CPU time, run queues, network counters, and interrupt totals answer different questions. If CPU time is saturated in application threads, changing device IRQ affinity is unlikely to solve the dominant cost. If interface drops rise while interrupts and receive processing lag, investigate the driver, queueing, and workload before forcing a CPU mask.

Look for temporal correlation. Compare interrupt deltas with the start of the workload, interface packet deltas, storage I/O, CPU idle time, and error logs. A counter that grows steadily during a normal workload can be expected. A counter that grows without corresponding device activity, or one handler consuming a disproportionate share while a relevant device stalls, deserves investigation but still requires driver-specific evidence.

Query interrupt affinity with cpuset

FreeBSD cpuset(1) accepts an interrupt identifier as a target using -x. First read the current mask for a numeric IRQ identifier that the system exposes:

cpuset -g -x 16

The example identifier is illustrative; use the actual interrupt number supported by the host. vmstat -i output formats and driver labels do not always map directly to an integer that can be supplied to cpuset, so verify the mapping from kernel messages, device attachment output, and the local cpuset(1) manual. If the identifier is ambiguous, do not change affinity.

The query shows an affinity mask, not a guarantee that the interrupt will generate work on every listed CPU or that one CPU’s interrupt count equals all work for the device. Interrupt controllers, MSI-X vectors, driver queues, CPU topology, and kernel scheduling affect where work runs. A NIC with several receive queues may have several vectors, each with different activity; a controller may use a shared line. Interpret each target in device context.

Change affinity only as a measured experiment

The same utility can set an IRQ’s CPU mask using -l and -x, but this is a privileged host-level change with hardware- and driver-dependent effects. Do not copy CPU numbers from another server. Before changing anything, record the current mask, available CPU set, IRQ identifier, workload, and baseline measurements:

cpuset -g -r
cpuset -g -x 16
vmstat -i

If testing is approved, use a maintenance window and set a mask that exists on this host. A syntax template is:

cpuset -l 2 -x 16
cpuset -g -x 16

This pins the selected IRQ target to CPU 2 only if that target and CPU are valid and the current policy permits the operation. It may increase locality for one workload but can also overload that CPU, interfere with a device driver, or worsen NUMA placement. Keep an out-of-band administration path and a documented rollback to the prior mask. The affinity may not persist across reboot; verify the mechanism and timing for the installed release before placing a command in boot configuration.

Do not confuse process affinity with interrupt affinity. cpuset -p PID targets a process, while cpuset -x IRQ targets an interrupt. Pinning the application to a CPU does not automatically pin its device handler, and pinning the handler does not reserve that CPU for the application. The best placement depends on cache locality, queue ownership, memory locality, and competing work.

Investigate device and driver symptoms first

If an expected device row is absent, check whether the device attached, whether its driver loaded, and whether the kernel logged a probe or resource failure. Use pciconf -lv, usbconfig, devinfo -rv, or the bus-appropriate inspection utility when relevant. Confirm that the hardware is supported by the release and that the driver reports the expected interface or controller. An absent interrupt row can also mean that the handler has not fired yet; use -a and a controlled device operation to test.

If one interrupt count is extremely high, compare its rate with the device’s actual activity and error counters. An interrupt storm typically means a device or handler is generating excessive events without productive progress, but a large legitimate packet or I/O rate can produce a large total. Check kernel messages, driver counters, queue drops, and a bounded reproduction. Avoid disabling interrupts or unloading a driver on a remote production host without a recovery plan.

If an IRQ mask query or change fails, record the exact error and check permissions, IRQ validity, supported operations, and CPU availability. Hardware may disallow certain affinity changes or the interrupt identifier may no longer match after reboot. Do not interpret a rejected change as proof that the driver is broken.

Use a hypothesis-driven test matrix

For a suspected imbalance, capture measurements under four states where possible: idle, normal production workload, controlled workload at expected volume, and recovery after load ends. Track per-device interrupt deltas, per-CPU utilization, scheduler run queues, device errors/drops, throughput, and application latency. Keep sample duration and workload constant.

Change only one variable at a time. If the hypothesis is that an interrupt is concentrated on an overloaded CPU, compare the original affinity with one safe alternative and restore the original mask after a bounded test. If CPU utilization changes but latency does not improve, the intervention did not meet its objective. If throughput falls or queue drops rise, roll back immediately and preserve the evidence.

For network devices, netstat -i and driver-specific statistics help relate interrupts to packet counts. For storage, pair the interrupt view with iostat/gstat and controller logs. For system-wide performance, use systat or vmstat rather than treating the interrupt report as a complete profiler. Hardware performance counters and DTrace can provide deeper evidence, but they add their own setup and measurement overhead.

Acceptance criteria

An interrupt investigation is complete when the device and handler are mapped, counters are compared over a known time window, correlated CPU and device evidence is collected, and the conclusion states what the data can and cannot show. An affinity change is accepted only if a repeatable workload improves a defined metric without increasing errors, drops, latency, or instability, and a rollback path has been exercised.

vmstat -i is an efficient starting point, not a verdict. Read its totals as cumulative accounting, combine them with device and CPU measurements, and resist turning a visible counter into a tuning target without evidence.

Related:

Sources: