Linux SLUB Allocator: Cache Statistics, Debug Flags, and Safe Triage

The Linux slab allocator serves small kernel objects from caches backed by page allocations. SLUB is the common allocator implementation on many kernels, but the exact allocator and debug capabilities depend on kernel configuration. A large cache is not automatically a leak: objects may be cached for reuse, held by legitimate kernel consumers, or retained as per-CPU and per-node allocator state.

Diagnose trends and object ownership before forcing reclaim or booting with global debugging. Slab debug options can change allocation layout, memory use, timing, and performance. A production system that begins failing under full slab debugging may be reacting to the instrumentation rather than reproducing the original workload faithfully.

Read cache state as a time series

Start with aggregate memory and slab views:

grep -E '^(Slab|SReclaimable|SUnreclaim):' /proc/meminfo
cat /proc/slabinfo
slabtop -o

slabtop availability and output vary. /proc/slabinfo is a snapshot; fields and cache names are implementation details that can change. Compare samples at a known interval and record kernel version, workload, and memory pressure. A high object count at one moment does not show whether it is growing, reclaimable, or actively used.

The slab counters in meminfo divide reclaimable and unreclaimable memory at a broad level. Cache-level information gives more detail but not the owner process for each object. Many kernel objects are not attributable to one userspace PID; network, filesystem, driver, and per-CPU state can all contribute.

Interpret active and total objects carefully. A cache can retain free objects within slabs for quick reuse. Pages cannot always be returned just because a process exited. Reclaimers and shrinkers may release eligible objects later, and some caches are intentionally retained to avoid allocation overhead.

Identify cache names and allocation paths

A cache name such as kmalloc-256 identifies a general size class, not one data structure or subsystem. Multiple call sites may allocate from it. Named caches can provide stronger hints, but the name still does not establish a leak. Inspect kernel source, trace data, or allocator tracking before blaming the first subsystem whose name resembles the cache.

When available and configured, SLUB debugfs tracking can expose allocation and free traces for selected caches. Allocation tracking consumes memory and CPU and may alter timing. It should be enabled only for a narrow target after collecting a baseline. Debugfs is not a stable userspace ABI; its presence and layout depend on kernel config and mount policy.

Kernel tracepoints in the kmem family can capture allocation and free events, but event volume can be enormous. Filter by a bounded process, cache, call site, or interval where supported. Store traces securely because stack addresses, process names, and kernel configuration can be sensitive. Disable instrumentation after the reproduction and verify the system returns to baseline behavior.

SLUB debugging modes and tradeoffs

The kernel’s slab debugging guide documents checks such as consistency verification, red zones, poisoning, tracking, and user tracking. Boot parameter syntax can target selected caches instead of enabling every check globally. Debugging may increase object size or slab order, reduce fast-path performance, and make a timing-sensitive defect disappear or change form.

Some debug options alter allocator behavior enough that the minimum slab order changes. A cache that normally packs many objects into one page may use a different layout when red zones or tracking metadata are enabled. This affects fragmentation and memory pressure. Capture boot parameters and allocator configuration with the test results so that debug evidence is not mistaken for ordinary production behavior.

If a corruption report names a cache and object offset, preserve the complete stack dump and surrounding bytes. A red-zone violation can identify an overwrite boundary, but the stack where corruption is detected may not be the writer. The responsible code could have written earlier, and an inactive poisoned object may have been reused after a lifetime bug.

Separate leak, retention, and corruption hypotheses

A leak hypothesis predicts that allocated objects rise without corresponding frees under a controlled workload and do not return after the owning activity stops. Retention predicts some reuse or reclaim behavior after pressure or cache shrink. Corruption predicts allocator consistency errors, red-zone damage, poison mismatch, or crashes. They require different evidence.

Compare a clean boot, idle baseline, workload growth, workload shutdown, and memory-pressure phase. Track per-cache object counts, total slab pages, reclaimable versus unreclaimable memory, and relevant kernel events. A cache that grows and plateaus can be a legitimate pool. A cache that grows linearly with each request may indicate a leak, but only allocation/free traces can identify the call path.

Tools such as kmemleak detect some orphaned allocations but do not replace slab accounting. They may produce false positives for memory reachable through unusual references and can miss leaks in retained caches. Use them in a controlled kernel test and validate each report against allocator ownership.

Safe debugging and performance validation

Do not switch on full slab debugging on a production server without a maintenance and rollback plan. Boot the same kernel with a narrow cache filter in a lab, reproduce the workload, collect the report, and compare throughput, latency, memory footprint, and failure timing with debugging disabled. If the corruption disappears under instrumentation, investigate timing and layout sensitivity rather than declaring success.

Before a kernel update or boot-parameter change, record the previous command line and ensure an alternate boot entry or console can restore it. A malformed or overly broad debug parameter can prevent boot or create a memory-starved system. Remove the parameter after evidence collection and verify it is absent from the next normal boot.

For live systems, begin with read-only /proc data and kernel logs. Avoid forcing shrinkers, dropping caches, or writing allocator debug controls as a generic “clear memory” operation. Those actions can degrade service, destroy evidence, or change a reproducer without identifying the owner.

Acceptance criteria

A useful report names the cache, object-count trend, bytes or pages, workload phase, allocator and kernel configuration, debug flags, allocation evidence, and observed impact. It distinguishes free objects retained for reuse from outstanding allocations and identifies whether a call site actually leaks or corrupts memory.

Test the candidate fix with debugging both enabled and disabled. Confirm that object counts return to an expected steady state after the workload, allocator warnings disappear, and performance remains within the service objective. Include a pressure test and long-duration soak; short boot-to-test runs can miss a slow leak.

SLUB is a high-performance object allocator with diagnostic modes, not a per-process memory report. Measure cache trends, narrow instrumentation, and interpret corruption evidence in the context of object lifetime and allocator layout before changing production boot settings.

Related:

Sources: