🚨 Linux Error Guide: 'load average: 14.52, 13.87, 12.31' — Find and Kill Runaway Processes

Overview

High CPU usage and an elevated load average are among the most common Linux performance problems. The issue appears when one or more processes consume nearly all available CPU cycles, causing the system to feel sluggish, starve other processes of scheduler time, and push the load average well above the number of logical CPUs.

The symptom you will typically see at the top of a top session looks like this:

top - 14:32:01 up 3 days,  2:15,  2 users,  load average: 14.52, 13.87, 12.31
Tasks: 312 total,   2 running, 310 sleeping,   0 stopped,   0 zombie
%Cpu(s): 98.7 us,  0.8 sy,  0.0 ni,  0.0 id,  0.4 wa,  0.0 hi,  0.1 si,  0.0 st
MiB Mem :  15872.0 total,    423.8 free,  14321.4 used,   1126.8 buff/cache

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
12345 app       20   0 2456789 1.2g   18456 R  99.9   7.8  42:18.03 python3

The three numbers after load average: represent 1-minute, 5-minute, and 15-minute exponential moving averages of the number of runnable or uninterruptible tasks on the run queue. On a host with 8 logical CPUs, a load average of 14.52 means the system is carrying nearly twice the work it can execute simultaneously — every CPU is saturated and tasks are queuing behind them.

The %Cpu(s) line breaks CPU time into segments: us (user-space code), sy (kernel/system calls), ni (niced processes), id (idle), wa (I/O wait), hi (hardware interrupts), si (software interrupts). A process pinned at 99.9% in the process list while id (idle) reads 0.0 confirms genuine CPU saturation, not an I/O problem.

Symptoms

• uptime or the top header shows a 1-minute load average greater than the CPU count, often by a large multiple.
• A specific PID appears pinned near 100% %CPU in top or htop and does not yield over multiple refresh cycles.
• Interactive sessions — SSH logins, shell commands — take unusually long to respond or hang momentarily.
• Application request latency spikes; health checks begin to fail or time out.
• Other processes that normally complete quickly are delayed because the CPU scheduler cannot give them time slices.
• System-level tools like ps, df, and find feel sluggish or block briefly before returning.
• /proc/loadavg reports values far above the logical CPU count over the 5- and 15-minute windows, indicating the problem is sustained rather than transient.

Common Root Causes

Infinite loop or tight spin in application code. A bug — a missing break condition, a recursive call that never terminates, or an exception handler that re-queues work without making progress — causes a process to consume 100% of one CPU core indefinitely.

Runaway background job. A cron job, systemd timer, or CI build that normally completes quickly gets stuck processing unexpectedly large or malformed input.

CPU-bound batch workload without resource limits. A legitimate workload such as compilation, model training, or video transcoding runs without nice, cpulimit, or cgroup CPU quotas and starves everything else on the host.

I/O wait inflating load average. Processes stuck in uninterruptible sleep (D state) waiting for a slow disk, NFS mount, or network filesystem count toward the load average even though they consume little CPU. A load average of 10 with wa (iowait) at 60% in top indicates a storage bottleneck, not a CPU bottleneck — a critical distinction before you kill anything.

Runaway kernel threads or IRQ handlers. A faulty driver or a device generating excessive interrupts can push %si (software interrupt) or %hi (hardware interrupt) high while no single user process appears obviously guilty.

Thread explosion. An application spawning hundreds of threads under load forces the kernel scheduler to context-switch constantly; the scheduler overhead itself drives CPU usage up while the actual work per thread stays low.

Diagnostic Workflow

Start with a quick sanity check using uptime to read the load averages and compare against the CPU count:

uptime
nproc --all          # number of logical CPUs (threads)
cat /proc/loadavg    # 1m 5m 15m, runnable/total tasks, last scheduled PID

If the 1-minute load exceeds nproc, something is running hotter than the host can handle. Open top interactively and press 1 to expand the per-core CPU breakdown; press P to sort by CPU:

top

For a non-interactive sorted snapshot of the top CPU consumers:

ps aux --sort=-%cpu | head -20

Use pidstat from the sysstat package to watch per-process CPU consumption over time and distinguish user-space (%usr) from kernel (%system) CPU:

pidstat -u 1 5           # 1-second intervals, 5 samples, all processes
pidstat -u 1 5 -p 12345  # focus on one specific PID

Use mpstat to see a per-core breakdown of user, system, iowait, and idle time across all CPUs simultaneously:

mpstat -P ALL 1 3

High iowait across all cores with moderate %usr points to a storage problem, not a runaway user-space process. Before killing anything, confirm what kind of saturation you have.

Check for D-state (uninterruptible sleep) tasks that inflate the load average without burning CPU:

ps aux | awk '$8 ~ /D/ {print $0}'

Once you identify a suspect PID, inspect what it is actually doing:

cat /proc/12345/status          # threads, VM state, memory
ls -l /proc/12345/exe           # which binary owns this PID
strace -p 12345 -c              # syscall summary; run 10 seconds then Ctrl-C
lsof -p 12345 | head -30        # open file descriptors and sockets

strace -c shows a count and percentage of each syscall type. A process spending 98% of its time in read, futex, or sched_yield with millions of calls is spinning in a loop rather than doing useful work.

Example Root Cause Analysis

Scenario: A Python ETL job starts processing a new data feed at 14:00. By 14:32 the server load average climbs to 14.52 on an 8-core machine. SSH sessions are sluggish and a web application running on the same host begins timing out.

Step 1 — confirm load vs CPU count:

$ uptime
 14:32:01 up 3 days,  2:15,  2 users,  load average: 14.52, 13.87, 12.31
$ nproc --all
8

Load average is 1.8× the CPU count and has been elevated for at least 15 minutes.

Step 2 — identify the offending process:

$ ps aux --sort=-%cpu | head -5
USER       PID %CPU %MEM    VSZ     RSS  TTY  STAT START   TIME  COMMAND
app      12345 99.9  7.8  2456789 1234567 ?   R    14:00  42:18  python3 /opt/etl/process.py

Step 3 — confirm user-space CPU rather than iowait:

$ pidstat -u 1 3 -p 12345
Average:   UID  PID  %usr  %system  %CPU  CPU  Command
Average:  1001 12345 99.8     0.1  99.9    3  python3

Nearly all CPU is user-space — this is an application bug, not I/O.

Step 4 — see what system calls it is making:

$ strace -p 12345 -c
^C
% time     seconds  usecs/call     calls    errors syscall
 98.14    3.847219           0  1243987           read
  1.86    0.072831           0    23497           lseek

Over 1.2 million read calls in 10 seconds. The process is reading one byte at a time in a tight loop, unable to reach EOF on a corrupt data record.

Step 5 — stop the process and restore service:

kill -15 12345       # SIGTERM — give it a chance to flush buffers
sleep 5
kill -9 12345        # SIGKILL if still running

After termination, load average returns to below 1.0 within two minutes. The input file is quarantined, the parser’s EOF handling is fixed, and the job is redeployed with a CPU limit:

nice -n 10 python3 /opt/etl/process.py

Prevention Best Practices

• Apply cgroup CPU limits to production workloads using systemd’s CPUQuota= directive or container resource limits so a runaway process cannot consume an entire host. For example, CPUQuota=200% restricts a service to 2 CPU cores regardless of how many are available.
• Run non-interactive batch jobs with nice -n 15 so they automatically yield to foreground workloads under contention.
• Set up monitoring alerts on load average exceeding 2× the CPU count sustained for more than 5 minutes. Prometheus node_exporter exposes node_load5 and node_cpu_seconds_total for this purpose.
• Use application-level timeouts and watchdog threads to detect and restart hung processes. systemd’s TimeoutStopSec and WatchdogSec directives handle this at the process supervisor level without custom code.
• Profile hot code paths with py-spy (Python), async-profiler (JVM), or Linux perf record before deploying CPU-intensive changes to catch algorithmic issues early.
• Enforce input size limits and rate limits on data-processing pipelines to prevent unexpectedly large or malformed payloads from triggering runaway loops.

Quick Command Reference

# Load average vs available CPUs
uptime && nproc --all
cat /proc/loadavg

# Top CPU consumers — sorted snapshot
ps aux --sort=-%cpu | head -20

# Per-process CPU breakdown over time (sysstat package)
pidstat -u 1 5
pidstat -u 1 5 -p <PID>

# Per-core CPU breakdown — spot iowait vs user vs sys
mpstat -P ALL 1 3

# Find D-state (uninterruptible sleep) tasks inflating load average
ps aux | awk '$8 ~ /D/ {print $0}'

# Inspect what a PID is doing
strace -p <PID> -c          # syscall summary (Ctrl-C to stop)
cat /proc/<PID>/status
ls -l /proc/<PID>/exe
lsof -p <PID> | head -30

# Graceful then forced kill
kill -15 <PID> && sleep 5 && kill -9 <PID>

# Run a job at reduced scheduling priority
nice -n 15 <command>

Conclusion

High CPU usage on Linux is almost always traceable to a specific PID within seconds of running top or ps aux --sort=-%cpu. The critical distinction is between true CPU saturation (high %usr or %sys) and load average inflation caused by I/O-waiting D-state tasks — mpstat -P ALL and pidstat -u make that separation clear without guesswork. Once you have identified the offending process, strace -c reveals whether it is spinning on syscalls, and kill -15 followed by kill -9 stops the immediate damage. Long-term, CPU quotas via cgroup/systemd, application watchdog timers, and load-average alerting prevent a single runaway job from degrading an entire host.