Reading /proc and /sys: The Kernel's Window into Userspace

Nearly every Linux diagnostic tool you’ve ever run — ps, top, lsof, free — is, underneath, just reading plain text files under /proc. There’s no special API most of these tools call; the kernel exposes its internal state directly as a filesystem, and userspace tools are mostly thin formatters around cat.

/proc: process and kernel state

/proc is a virtual filesystem — nothing under it exists on disk — generated on the fly by the kernel each time it’s read. Every running process gets a numbered directory, /proc/<pid>/, exposing its state in enormous detail:

cat /proc/1234/status        # human-readable process state summary
cat /proc/1234/cmdline       # the exact command line it was started with
ls -la /proc/1234/fd/        # every open file descriptor, as symlinks
cat /proc/1234/maps          # its memory mappings

/proc/<pid>/fd/ is particularly useful in practice — each entry is a symlink to whatever the process has open, which is exactly how lsof (and fuser) determine which process holds a given file or socket without any special kernel API beyond reading these symlinks.

ls -la /proc/1234/fd/
# lrwx------ 1 user user 64 ... 3 -> /var/log/app.log
# lrwx------ 1 user user 64 ... 4 -> socket:[88213]

System-wide state under /proc

Beyond per-process directories, /proc also exposes system-wide kernel state that most monitoring tools ultimately read from directly:

cat /proc/meminfo       # what `free` parses and reformats
cat /proc/loadavg       # what `uptime` reads
cat /proc/cpuinfo       # per-core CPU details
cat /proc/interrupts    # interrupt counts per CPU, per device

/proc/net/ similarly exposes networking state — /proc/net/tcp lists every TCP socket’s state in a raw, numeric form that tools like ss and netstat parse and present more readably.

Tuning the kernel: /proc/sys and sysctl

A specific subtree, /proc/sys/, is writable and controls tunable kernel parameters directly — writing a new value takes effect immediately, no reboot required:

cat /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/ip_forward

sysctl is simply a friendlier interface to this same tree, translating dotted names to paths under /proc/sys:

sysctl net.ipv4.ip_forward
sysctl -w net.ipv4.ip_forward=1

# /etc/sysctl.conf — persisted across reboots, applied via sysctl -p
net.ipv4.ip_forward = 1
vm.swappiness = 10

sysctl -a dumps every tunable currently exposed, and the mapping is exact: net.ipv4.ip_forward is nothing more than a friendly name for /proc/sys/net/ipv4/ip_forward.

/sys: a structured view of the device model

Where /proc grew organically over decades and mixes process info with fairly arbitrary kernel internals, /sys (sysfs) was designed later, specifically to expose the kernel’s internal device model in a consistent, structured way — every device, driver, and bus the kernel knows about, represented as a directory tree of attributes and symlinks.

ls /sys/class/net/            # every network interface the kernel knows about
cat /sys/class/net/eth0/speed
cat /sys/block/sda/size        # device size, in 512-byte sectors

Symlinks under /sys encode the actual hardware topology — /sys/class/net/eth0/device links to the PCI device backing that interface, and /sys/bus/pci/devices/ lists every PCI device the kernel enumerated, letting you walk from a network interface name to its exact physical bus location without any additional tooling.

readlink -f /sys/class/net/eth0/device

Writable sysfs attributes: runtime hardware control

Like /proc/sys, many /sys attributes are writable and take effect immediately — CPU frequency scaling governors, LED brightness, USB autosuspend behavior, and more:

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

Why both still exist

/proc and /sys overlap in spirit but not in scope: /proc remains the home for process-specific state and a long tail of legacy kernel internals that predate sysfs’s stricter conventions, while /sys is where the modern device and driver model — and any newly added kernel subsystem’s tunables — is expected to live. In practice, knowing that everything the kernel is willing to tell you is sitting in one of these two trees, readable with nothing more than cat, turns a huge class of “how do I find out X about this running system” questions into “which file under /proc or /sys has X,” which is very often just a find or grep away.