🔬 Exploring /proc and /sys: The Linux Admin's Window Into the Kernel

Every Linux admin tool you trust — top, ps, free, lsof, ss — is, under the hood, reading files in /proc and /sys. These virtual filesystems expose the kernel’s live state as plain files you can cat, and once you know your way around them you can answer questions no canned tool will, and tune things on the fly. After 25 years these directories are still where I go when the friendly tools don’t tell me enough.

/proc vs /sys, briefly

• /proc — originally process info (one numbered directory per PID), it grew to hold system-wide kernel state and tunables. Older, less structured.
• /sys (sysfs) — a newer, more organized tree exposing the kernel’s device and driver model: buses, devices, modules, and their attributes.

Neither lives on disk. Reading a file pulls the value live from the kernel; writing one (where permitted) changes kernel state immediately.

Per-process forensics under /proc/PID

Every running process has a /proc/<pid>/ directory. This is where you investigate a process that’s misbehaving and won’t tell you why.

ls -l /proc/1234/exe          # symlink to the actual binary on disk
cat /proc/1234/cmdline | tr '\0' ' '   # exact command line, args and all
cat /proc/1234/environ | tr '\0' '\n'  # the process's environment variables
ls -l /proc/1234/fd            # every open file descriptor (sockets, files, pipes)
cat /proc/1234/status          # state, memory, UID, threads, capabilities
cat /proc/1234/limits          # the actual rlimits in effect for this PID

Two of these have saved me repeatedly. /proc/<pid>/fd shows exactly what a process has open — invaluable for “what’s holding this deleted file and eating disk?” (a deleted-but-open file shows as (deleted) in the symlink). And /proc/<pid>/limits shows the real limits a running process has, which is the only reliable way to confirm an ulimit change actually took effect.

# Find which process is pinning a deleted file that's eating disk
sudo ls -l /proc/*/fd 2>/dev/null | grep '(deleted)'

Memory and mapping detail

cat /proc/meminfo             # the source of truth `free` reads
cat /proc/1234/smaps_rollup   # per-process memory breakdown: RSS, PSS, swap
cat /proc/1234/oom_score      # how likely the OOM killer is to pick this PID

smaps_rollup gives you PSS (proportional set size), which fairly attributes shared memory — the honest number when you’re hunting a memory hog among processes that share libraries.

System-wide tunables: /proc/sys and sysctl

/proc/sys/ is the kernel’s tunable knobs, and sysctl is just a friendly front-end to it. These are the same thing:

sysctl net.ipv4.ip_forward
cat /proc/sys/net/ipv4/ip_forward      # identical value

Change one live:

echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward
# equivalently: sudo sysctl -w net.ipv4.ip_forward=1

Live writes are lost on reboot. Persist them in /etc/sysctl.d/*.conf. Useful knobs to know:

cat /proc/sys/vm/swappiness            # how eagerly to swap
cat /proc/sys/fs/file-max              # system-wide open-file limit
cat /proc/sys/kernel/pid_max           # max PID before wraparound
cat /proc/sys/net/core/somaxconn       # max accept queue (raise for busy servers)

somaxconn in particular: if a high-traffic service drops connections under load, this kernel ceiling is a frequent culprit no application config will reveal.

Pressure stall information: the modern saturation signal

A newer and underused part of /proc is PSI — pressure stall information — which tells you how much time tasks spent stalled waiting for CPU, memory, or I/O:

cat /proc/pressure/cpu
cat /proc/pressure/memory
cat /proc/pressure/io

This is a better “is the system saturated?” signal than load average, because it directly measures contention. Rising some/full averages mean real pressure — alert on these.

The device tree under /sys

/sys is where you inspect and tweak hardware. A few high-value spots:

# Block device details
cat /sys/block/sda/queue/rotational    # 1 = spinning disk, 0 = SSD
cat /sys/block/sda/queue/scheduler     # active I/O scheduler, brackets show current

# Change the I/O scheduler live
echo mq-deadline | sudo tee /sys/block/sda/queue/scheduler

# Network interface state and speed
cat /sys/class/net/eth0/operstate      # up / down
cat /sys/class/net/eth0/speed          # link speed in Mbps

# CPU frequency governor
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

Switching the CPU governor to performance via this path is a common quick win for latency-sensitive boxes that ship defaulted to powersave:

echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Thermal, power, and module info

cat /sys/class/thermal/thermal_zone0/temp   # millidegrees C
cat /sys/class/power_supply/BAT0/capacity    # laptop battery %
ls /sys/module/<name>/parameters/            # live params of a loaded module

The thermal zones are how you confirm “is this box throttling because it’s overheating?” without vendor tooling.

A few rules to keep you safe

• Reading is always safe. Writing changes live kernel state instantly — know what a knob does first.
• Anything you write here is not persistent. Use /etc/sysctl.d/ for tunables and the proper config for device settings.
• Don’t script blind loops over /proc/*/ on huge process counts without care; it’s cheap but not free.

Where AI helps

The challenge with /proc and /sys isn’t reading a file — it’s knowing which of thousands of entries answers your question, and interpreting dense output like smaps_rollup or PSI. Describe the symptom to a model (“connections dropping under load, where in /proc should I look?”) and it’ll point you at somaxconn, the accept queue, and PSI. I keep a few Linux admin prompts for exactly this kind of guided exploration.

/proc and /sys are the kernel with its hood open. Every monitoring tool is a thin wrapper over them — learn to read the files directly and you’ll diagnose things the wrappers never surface.

Generated commands and configs are assistive, not authoritative. Always verify against your own systems before applying changes in production.