🚨 Linux Error Guide: 'watchdog: BUG: soft lockup - CPU stuck' — Diagnose Kernel Stalls

Overview

The kernel’s soft-lockup detector fires when a CPU stays in kernel mode without scheduling for longer than the threshold, and dumps a backtrace to the console and dmesg:

watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kworker/3:1:412]

It is typically followed by register state and a call trace that points at where the CPU was stuck:

Modules linked in: nf_conntrack ...
CPU: 3 PID: 412 Comm: kworker/3:1 Tainted: G           O
Call Trace:
 native_queued_spin_lock_slowpath+0x1a0/0x1e0
 _raw_spin_lock+0x1c/0x30
 ...

A “soft lockup” means a task hogged a CPU in the kernel without yielding (unlike a hard lockup, where interrupts stopped). The machine often keeps running, but the stall signals a spinlock storm, a misbehaving driver, storage that stopped responding, or — on VMs — the hypervisor starving the guest of CPU.

Symptoms

• dmesg shows one or more soft lockup - CPU#N stuck for Ns messages, sometimes repeating.
• Brief hangs, dropped packets, or latency spikes coinciding with the messages.
• A specific kworker, application, or IRQ thread named in the trace.
• On cloud VMs, the stalls line up with high st (steal) time.
• In severe cases the soft lockup escalates to a hard lockup or panic.

Common Root Causes

• Spinlock contention. native_queued_spin_lock_slowpath in the trace = many CPUs fighting for one lock (often network/conntrack or a subsystem under extreme load).
• CPU steal on virtual machines. The hypervisor didn’t schedule the vCPU for seconds, so the guest thinks its own CPU is stuck.
• A blocking driver / IRQ handler. A NIC, storage, or GPU driver spinning or waiting on hardware that stopped responding.
• Storage stalls. A path/device (SAN, NVMe, iSCSI) hanging keeps a CPU busy-waiting in the I/O path.
• Out-of-memory / reclaim storms. Heavy direct reclaim or a memory leak keeping a CPU pinned.
• Faulty hardware or firmware. Bad RAM, thermal throttling, or buggy microcode, sometimes with a Tainted flag from an out-of-tree module.

Diagnostic Workflow

Pull the full lockup events with surrounding context — the call trace names the culprit subsystem:

dmesg -T | grep -A25 -i 'soft lockup'
journalctl -k -b | grep -A25 -i 'soft lockup'

On a VM, rule in/out steal time first — it’s the most common cause and needs no kernel forensics:

vmstat 1 5           # watch the 'st' column
mpstat -P ALL 1 5    # %steal per CPU

Look at what was running and whether storage or the network was stalled:

dmesg -T | grep -iE 'hung|blocked|nfs|scsi|nvme|task .* stuck'
pidstat -t 1 5       # per-thread CPU; find the pegged kworker/app
cat /proc/loadavg

Check the current watchdog thresholds and whether the kernel is tainted:

sysctl kernel.watchdog kernel.watchdog_thresh kernel.softlockup_panic
cat /proc/sys/kernel/tainted        # non-zero = out-of-tree/proprietary module loaded

Example Root Cause Analysis

A fleet of KVM guests began logging watchdog: BUG: soft lockup - CPU#1 stuck for 22s! several times a day, with brief application freezes but no crash.

The call traces were inconsistent — different tasks each time — which argued against a single buggy driver. Running mpstat -P ALL 1 during an event showed %steal spiking above 40% on the affected vCPU exactly when the message appeared. The guests weren’t stuck at all; the oversubscribed hypervisor simply wasn’t scheduling their vCPUs for tens of seconds, and the guest’s soft-lockup detector interpreted the gap as its own CPU hanging.

The real fix was on the host: reduce vCPU oversubscription and add CPU reservations for these guests. As an interim measure the team raised kernel.watchdog_thresh so transient steal no longer tripped the detector, while explicitly keeping softlockup_panic=0 so a genuine future lockup would log rather than reboot the box before they could capture it.

Prevention Best Practices

• On VMs, monitor and alarm on steal time; right-size vCPU allocation and avoid heavy oversubscription for latency-sensitive guests.
• Keep kernels, NIC/storage drivers, and CPU microcode current — many soft lockups are known driver/firmware bugs already fixed upstream.
• Avoid loading unsupported out-of-tree modules on critical hosts; a tainted kernel narrows vendor support and is a common lockup source.
• Ensure storage paths fail fast (multipath timeouts, sane nvme/iscsi timeouts) so a dead path doesn’t pin a CPU indefinitely.
• Configure kdump so a soft lockup that escalates to a panic produces a crash dump for root-cause analysis instead of a silent reboot.

Quick Command Reference

# Capture the events and traces
dmesg -T | grep -A25 -i 'soft lockup'
journalctl -k -b -p err

# Rule in/out CPU steal (VMs)
mpstat -P ALL 1 5
vmstat 1 5

# Inspect thresholds (defaults: watchdog_thresh=10 -> ~20s report window)
sysctl kernel.watchdog_thresh kernel.softlockup_panic

# Cautiously relax the detector (interim only, VMs with steal)
sudo sysctl -w kernel.watchdog_thresh=30

# Ensure crash capture is armed
systemctl status kdump
cat /proc/sys/kernel/tainted

Conclusion

A soft lockup - CPU stuck means a CPU spent too long in the kernel without yielding — read the call trace to tell a spinlock storm or blocking driver from the very common VM case of CPU steal. On virtual machines, check %steal before anything else; on bare metal, follow the trace to the stalled subsystem and update the driver, firmware, or storage timeouts. Tune watchdog_thresh only as an interim measure, keep kdump armed, and fix the underlying contention rather than silencing the detector.