Linux Production Troubleshooting

The host-level skills every other stack sits on: disk, filesystem, memory, processes, and the commands that expose what a box is actually doing.

0 of 4 modules complete

Who it’s for: Anyone operating Linux servers who wants a repeatable triage method instead of guesswork.
Prerequisites: Basic shell familiarity.

Skills you’ll build

✓Read disk/inode/memory pressure quickly
✓Trace a process’s resource use
✓Use the right diagnostic command for the symptom

4 modules · self-paced

Module 1

Command-line fluency for diagnosis

Know the safe, high-signal commands (and the destructive ones to flag) before you need them under pressure.

Linux Commands core guide Browse the Linux stack
Diagnostic commands run in order — each one narrows the fault
1. Check system load and run queue
```
uptime && vmstat 1 5
```
  Load above the CPU count with high `r` in vmstat means CPU saturation; high `b` and `wa` means the box is blocked on I/O, not compute.
2. List failed systemd units
```
systemctl --failed --no-pager
```
  Any unit here is a service that tried to start and gave up. Empty output is the healthy case.
3. Read the most recent errors
```
journalctl -p err -b --no-pager | tail -50
```
  Priority-error entries since boot. Look for the FIRST error in a burst — later ones are usually consequences.
4. Test DNS resolution
```
resolvectl query example.com || dig +short example.com
```
  No answer, or a long delay before one, points at /etc/resolv.conf, the systemd-resolved stub, or an upstream resolver.
5. Check routes and default gateway
```
ip route show && ip -brief address
```
  A missing default route explains "Network is unreachable" instantly. Confirm the interface is UP and holds the address you expect.
6. List listening ports
```
ss -tulpn
```
  Confirms whether the service is actually bound, and on which address. Bound to 127.0.0.1 when you expected 0.0.0.0 is a very common cause of "connection refused".
Module 2

Disk and filesystem errors

Diagnose "no space left on device" (bytes vs inodes), read-only remounts, and mount failures.

Linux disk & filesystem errors hub Linux error guides
Diagnostic commands run in order — each one narrows the fault
1. Check free space and inodes
```
df -h && df -i
```
  A filesystem can be 40% full by bytes and 100% full by inodes — both produce "No space left on device". Always check both.
2. Find what is consuming the space
```
du -xh --max-depth=1 / 2>/dev/null | sort -rh | head -20
```
  `-x` keeps it on one filesystem so you do not walk network mounts. Work down the tree from the largest entry.
3. Find deleted-but-open files
```
lsof +L1 2>/dev/null | head -20
```
  Space that df reports as used but du cannot find is usually a deleted file still held open by a process. Restarting that process reclaims it.
4. Check for read-only remounts
```
mount | grep -E "\\bro\\b" && dmesg -T | grep -iE "remount|ext4|xfs|i/o error" | tail -20
```
  A filesystem remounted read-only means the kernel hit an I/O error and protected itself. Treat it as a hardware/storage problem until proven otherwise.
Exercise

A service is failing writes but `df -h` shows free space. Determine whether you’re out of inodes, hitting a read-only filesystem, or blocked by permissions.
Open in Workspace →
Module 3

Processes, memory, and resource pressure

Find the process behind CPU/memory pressure and understand OOM behavior.

Linux Commands core guide DevOps Tools core guide
Diagnostic commands run in order — each one narrows the fault
1. Top memory and CPU consumers
```
ps aux --sort=-%mem | head -15
```
  Identifies the process actually holding the memory, rather than the one that happened to fail first.
2. Check for OOM kills
```
journalctl -k --no-pager | grep -i "killed process" | tail -20
```
  Each line names the victim and its RSS. A repeatedly-killed process is under-provisioned or leaking.
3. Inspect memory pressure and swap
```
free -h && cat /proc/pressure/memory 2>/dev/null
```
  PSI `some avg10` above ~10 means real memory stall, even when free memory looks acceptable.
4. Check open file descriptor limits
```
ulimit -n && cat /proc/sys/fs/file-nr
```
  "Too many open files" is almost always the per-process soft limit, not the system-wide one. Compare both.
Module 4

Run a host incident in the workspace

Triage a slow/unhealthy host end-to-end and export the summary.

Open the Troubleshooting Workspace

Exercise

A host is slow and one service keeps restarting. Triage disk, memory, and the failing unit; record findings and root cause.
Open in Workspace →

Free DevOps tools

Command-line fluency for diagnosis

Disk and filesystem errors

Processes, memory, and resource pressure

Run a host incident in the workspace