🚨 Linux Error: 'A stop job is running for...' — Cause, Fix, and Troubleshooting Guide

Summary

A stop job is running for <unit> (<n>s / 1min 30s) is the countdown systemd prints during shutdown or reboot while it waits for a unit to stop. It is not an error in itself — it means a service ignored SIGTERM and systemd is now waiting out TimeoutStopSec before it sends SIGKILL. A machine that takes 90 seconds or 5 minutes to reboot almost always has one specific unit that refuses to die cleanly.

Common Symptoms

• Reboot/poweroff hangs on A stop job is running for <unit> with a rising timer.
• Shutdown consistently pauses ~1m30s (the default DefaultTimeoutStopSec) or ~5min.
• The same unit name is named every time.
• After the timeout the machine finally powers off (systemd SIGKILLed it).
• Journals show Stopping <unit>... with a long gap before Stopped.

Most Likely Causes of the ‘A stop job is running for…’ Error

The A stop job is running for... message means a unit did not exit within its stop timeout. Common production causes, most frequent first:

1. The service ignores or is slow to handle SIGTERM — no clean shutdown handler, so it runs until the timeout forces SIGKILL.
2. A hung ExecStop= command that blocks (e.g. a graceful drain that never returns).
3. Long, legitimate cleanup — flushing a large write buffer, closing many connections, or a database checkpoint.
4. Network/remote-filesystem unmounts stalling (NFS server gone, so umount blocks).
5. A wrong KillMode=/Type= so systemd is waiting on the wrong PID and never sees the real process exit.
6. A stuck user session or user@<uid>.service manager holding processes open.

Quick Triage

# After the slow reboot, find which unit ate the time
journalctl -b -1 -u systemd-shutdown --no-pager
journalctl -b -1 | grep -iE 'stop job|Stopping|Stopped|Timed out'
systemd-analyze blame                       # startup side, but flags heavy units

journalctl -b -1 reads the previous boot, which is where the slow shutdown was logged.

Diagnostic Commands

# See the effective stop timeout and kill behaviour for the culprit
systemctl show <unit> -p TimeoutStopUSec -p KillMode -p ExecStop -p Type
systemctl cat <unit>
systemctl show -p DefaultTimeoutStopUSec    # global default (usually 90s)

TimeoutStopUSec is how long systemd waits before SIGKILL; if it equals the shutdown pause, that unit is your hang.

# Watch the actual stop happen (in a test window)
journalctl -u <unit> -b
systemctl list-jobs                          # jobs still running during transition

# For a stalled unmount during shutdown
findmnt --types nfs,nfs4,cifs                # remote mounts that can block umount
journalctl -xb -1 | grep -iE 'umount|unmounting|nfs'

If a remote filesystem’s server is unreachable, umount blocks and systemd waits the full timeout — the fix is _netdev/nofail and forcing the unmount.

Fix / Remediation

1. Make the service handle SIGTERM. The correct fix is in the app — trap the signal and exit promptly. If it needs a different stop signal, declare it:

   sudo systemctl edit <unit>

   [Service]
   KillSignal=SIGINT
   KillMode=mixed

2. Set a realistic per-unit stop timeout so a genuinely slow (but bounded) shutdown is not cut off, or a stubborn one is killed sooner:

   [Service]
   TimeoutStopSec=20

   sudo systemctl daemon-reload

3. Fix a hanging ExecStop= — make the drain command bounded (add its own timeout 15 ...) or remove it if SIGTERM already stops the process cleanly.

4. Lower the global default on hosts that must reboot fast (affects every unit without its own value):

   # /etc/systemd/system.conf
   DefaultTimeoutStopSec=30s

   sudo systemctl daemon-reexec

5. For blocking remote unmounts, add nofail,_netdev in /etc/fstab and ensure the network stops after the mounts.

Warning: Do not set TimeoutStopSec=0 (disables the timeout) or slash DefaultTimeoutStopSec to a couple of seconds globally — you will SIGKILL databases mid-flush and risk data corruption. Give real cleanup enough time; only shorten it for services you know stop instantly.

Validation

sudo systemctl daemon-reload
sudo systemctl stop <unit>                   # time a manual stop
systemctl show <unit> -p TimeoutStopUSec     # confirm new value
# Then, in a maintenance window:
sudo reboot
journalctl -b -1 | grep -i 'stop job'        # expect no long-running stop job

Prevention

• Implement graceful SIGTERM handling in every long-running service so it exits well under the timeout.
• Set a per-unit TimeoutStopSec= matched to real cleanup time instead of relying on the 90s default.
• Use KillMode=mixed and the correct KillSignal= so systemd signals the right process.
• Mark remote filesystems nofail,_netdev so a dead server cannot stall shutdown.
• Track reboot/shutdown duration and review journalctl -b -1 for slow stops after changes. See the Linux admins guides.

Related Errors

• Job timed out
• Device or resource busy
• Failed to start unit
• Dependency failed for /
• Start request repeated too quickly

Final Notes

A stop job is running for... is systemd being patient with a service that will not stop. Read journalctl -b -1 to name the exact unit, check its TimeoutStopUSec, and fix the real cause — usually missing SIGTERM handling or a hung ExecStop. Tune TimeoutStopSec per unit rather than gutting the global default, so fast reboots never come at the cost of corrupting data mid-shutdown.

Want faster Linux incident response? Use DevOps AI Toolkit to turn production errors into clear diagnostics, remediation steps, and reusable runbooks.