Reading /proc and /sys: The Kernel's Window into Userspace

procfs and sysfs expose kernel-generated objects through filesystem operations, but they are not ordinary disk trees and not the only APIs used by diagnostic tools. Modern tools also use netlink, ioctls, system calls, BPF, debugfs, tracefs, and libraries. Reads can race with live state, reveal secrets, block, or return namespace-specific views; writes can alter the running kernel or hardware immediately.

Treat both mounts as privileged, volatile interfaces with documented ABI—not convenient files to crawl recursively.

Confirm the mounts and observer context

findmnt -t proc,sysfs
readlink /proc/self/ns/{pid,user,mnt,net}
cat /proc/self/cgroup

Containers, chroots, and services can receive restricted/read-only/subset mounts. A path visible in one mount or PID namespace may describe a different task/device view than the host. Record who observed it, from where, and when.

PID directories are race-prone identities

/proc/<pid> exists while that task is represented and accessible. The process can exit between open() calls, and the numeric PID can later be reused. For automation requiring stable process identity, use pidfds where supported instead of repeatedly trusting a number.

Capture start time and namespace/cgroup alongside PID. Do not attribute a later status file to the process named in an old alert solely because the number matches.

Process files have different encodings

status is line-oriented summary text; stat is a positional format whose command name can contain spaces/parentheses; cmdline and environ are NUL-separated; maps/smaps describe mappings with access controls and potentially sensitive paths.

tr '\0' ' ' </proc/"$pid"/cmdline; printf '\n'
sed -n '1,80p' /proc/"$pid"/status
ls -l /proc/"$pid"/fd

Command lines and environments can contain passwords, tokens, keys, URLs, or customer data. Avoid logging them wholesale. Access is governed by ownership, ptrace checks, Yama/LSM policy, capabilities, and procfs mount options.

File-descriptor links need careful interpretation

/proc/<pid>/fd/N links can name files, socket:[inode], pipe:[inode], anonymous inodes, deleted paths, or objects inaccessible to the observer. Opening an fd link can duplicate access to the underlying object if permissions allow; this is not a harmless metadata operation.

lsof uses procfs heavily on Linux but also performs device/socket mapping and permission-aware logic. A manual ls is not a complete substitute, especially across mount/network namespaces.

`/proc/net` follows the network namespace

On current Linux, /proc/net points to /proc/self/net, so its socket/interface tables reflect the reader’s network namespace. To inspect another process, use /proc/<pid>/net with appropriate authorization or enter the namespace through an approved diagnostic path.

Prefer ss/netlink for supported socket queries. Raw table formats are kernel ABIs with encoded addresses/states and should not be parsed by column guesses.

System-wide proc data is still contextual

/proc/meminfo, loadavg, pressure, interrupts, zoneinfo, and vmstat expose kernel state, but container runtimes may virtualize or leave host-like values visible. Host memory totals do not necessarily reflect a cgroup limit; combine proc data with cgroup v2 files.

Counter samples need intervals and CPU hotplug/wrap awareness. A single interrupt or VM counter value rarely proves a fault.

`/proc/sys` is a live control plane

The sysctl hierarchy maps many dotted names to files, but not every proc file is a sysctl and not every kernel control lives there. Read narrowly:

sysctl net.ipv4.ip_forward
sysctl vm.overcommit_memory

sysctl -a can be noisy, slow, permission-denied, and expose sensitive policy. Writing with sysctl -w or redirection changes runtime state and can break routing, memory behavior, security, or availability. Capture old value, authoritative documentation, dependency impact, and rollback before any change.

Persistence is directory- and manager-specific

On systemd-based distributions, systemd-sysctl reads ordered sysctl.d files from vendor/runtime/local directories with precedence rules. /etc/sysctl.conf may be supported through compatibility tooling, but sysctl -p reads a specified file and is not proof that boot-time precedence matches it.

Use /etc/sysctl.d/90-local-purpose.conf only according to distribution policy, choose one owner per key, and inspect effective ordering:

systemd-analyze cat-config sysctl.d

Do not write all current sysctls into a persistence file; many are dynamic, namespace-specific, hardware-dependent, or unsafe across kernel updates.

sysfs represents kobjects and relationships

sysfs exports devices, drivers, buses, classes, firmware, modules, and other kernel objects. Class directories contain convenient views/symlinks into physical topology:

readlink -f /sys/class/net/eth0/device
udevadm info --query=all --path=/sys/class/net/eth0

Names such as eth0, sda, and hwmon0 can change. Use stable attributes and topology appropriate to the task, and remember that a symlink target can disappear during hot unplug.

sysfs attributes follow a documented ABI

Many text attributes present one value per file, but binary attributes and exceptions exist. Units are attribute-specific: a block-device size commonly counts 512-byte sectors independent of physical sector size; temperatures and power controls may use scaled integers.

Read the relevant file under the kernel’s Documentation/ABI and subsystem documentation before interpreting or writing. Never infer units from a plausible number.

Writable sysfs is hardware control

Writing a governor, device authorization flag, queue setting, bind/unbind control, reset attribute, LED, or power-management value can disrupt hardware or storage immediately. Paths vary with drivers and hotplug, and values may not persist.

Do not demonstrate with echo performance > ... on an unknown host. Use distribution power-management/service interfaces, a maintenance window, thermal/power monitoring, old-value capture, and a verified recovery console.

Virtual files can block or have side effects

A read may invoke driver code, generate data, wait on locks, or clear/read-to-acknowledge hardware state for subsystem-specific attributes. Recursive find, grep -R, backup, indexing, or antivirus scans over /proc and /sys can race, flood errors, hang, or expose data.

Query exact documented files. Exclude virtual filesystems from ordinary backups; recreate state through configuration, not restoration of captured pseudo-files.

Permissions and LSMs intentionally hide data

procfs hidepid/subset options, ptrace restrictions, SELinux/AppArmor, kernel lockdown, user namespaces, and capabilities limit observation or writes. An empty/denied result is part of the security model, not a reason to remount proc broadly or grant CAP_SYS_ADMIN.

For support collection, run the smallest approved helper and redact paths, addresses, command lines, environment, kernel pointers, device serials, and tenant process data.

Build a reproducible inspection bundle

Record kernel and distribution version, timestamp/timezone, observer UID/capabilities/namespaces/cgroup, mount options, exact files/commands, units, and sampling interval. Copy only necessary text output, not the virtual trees.

The investigation is trustworthy when each field has documented semantics, volatile identities are correlated, namespace scope is explicit, secrets are minimized, and no diagnostic write changed the system being measured.

Related:

Sources: