🚨 Linux Error Guide: 'Resource temporarily unavailable' — Clear Stale Locks

Overview

When a program uses a lock to guarantee that only one instance runs at a time, a crash can leave that lock in place with no live process behind it. The next start then fails in one of two recognizable ways:

flock: cannot lock: Resource temporarily unavailable
another instance is already running (PID 4821)

The first message comes from the kernel returning EAGAIN (errno 11) when a non-blocking flock(2) or fcntl(2) lock request cannot be granted because another open file description still holds the lock. The second is an application-level message printed after a program reads a PID or lock file and refuses to continue.

The tricky part is telling a genuinely running instance apart from a stale lock left by a crash or an unclean shutdown. This guide shows how to inspect PID files, verify whether the recorded process is actually alive, test advisory locks safely, and clear only the locks you have proven to be stale, all while letting systemd manage service lifecycle where possible.

Symptoms

• A service refuses to start, logging Resource temporarily unavailable or another instance is already running.
• A PID file such as /run/app/app.pid exists but the PID it names is gone or belongs to an unrelated process.
• systemctl start fails and the unit sits in a failed state after a crash or a hard power loss.
• A cron job wrapped in flock silently exits because the previous run’s lock was never released.
• The lock file’s modification time is old, from before the last reboot.

Common Root Causes

1. Stale PID file after a crash. The process died without removing /run/app/app.pid, so the startup check sees the file and assumes a peer is alive.
2. Advisory lock held by a lingering process. An flock/fcntl advisory lock is still held by a process (or a child that inherited the file descriptor) that has not exited.
3. Unclean shutdown. A power loss or SIGKILL prevented normal cleanup, leaving lock files in /run or /var/run.
4. Reused PID. The PID recorded in the file has since been recycled by the kernel and now names a different, unrelated process, fooling a naive kill -0 check.
5. systemd PIDFile mismatch. A unit with a PIDFile= directive points at a path the daemon wrote incorrectly, leaving systemd and the app disagreeing about liveness.

Diagnostic Workflow

Start by reading the PID file and asking whether that process still exists. kill -0 sends no signal but succeeds only if the PID is alive and signalable by you:

cat /run/app/app.pid
ps -p "$(cat /run/app/app.pid)" -o pid,ppid,user,comm,args
kill -0 "$(cat /run/app/app.pid)" && echo alive || echo dead

If the PID is alive, confirm it is really your application and not a recycled PID by inspecting its process directory:

ls -l /proc/"$(cat /run/app/app.pid)"/exe
cat /proc/"$(cat /run/app/app.pid)"/cmdline | tr '\0' ' '; echo

To find out who holds an advisory lock, list system-wide locks and identify which PID owns the file:

sudo lslocks
sudo fuser -v /run/app/app.lock

Test the lock directly without blocking. flock -n tries once and exits non-zero if the lock is held; if it succeeds, the lock was free and therefore stale:

flock -n /run/app/app.lock -c true && echo "lock is free (stale)" || echo "lock is held"

Check how old the lock file is; a timestamp older than the last boot is a strong stale signal:

stat -c '%y %n' /run/app/app.lock
uptime -s

Example Root Cause Analysis

After an unplanned reboot, an operator found that a batch importer would not start. Its wrapper script logged flock: cannot lock: Resource temporarily unavailable, which looked, at first glance, like a second copy was already running.

The PID file check said otherwise:

cat /run/importer/importer.pid   # printed 4821
kill -0 4821 && echo alive || echo dead   # printed dead

PID 4821 no longer existed, so nothing living held anything. To be certain the advisory lock file itself was free, the operator tested it non-blockingly:

flock -n /run/importer/importer.lock -c true && echo free || echo held

It printed free, and stat showed the lock file’s mtime predated the current boot time from uptime -s. Together these proved the lock was a leftover from the process that died in the crash, not evidence of a running peer. The operator removed the confirmed-stale files and reset the failed unit so systemd would allow a clean start:

sudo rm -f /run/importer/importer.pid /run/importer/importer.lock
sudo systemctl reset-failed importer.service
sudo systemctl start importer.service

The importer started immediately. The durable fix was to stop hand-rolling the PID check and let systemd own the lifecycle instead (see below), so a crash could never again strand a lock file.

Prevention Best Practices

• Wrap single-instance jobs in flock(1). Instead of parsing PID files, run flock -n /run/app/app.lock -c 'your-command'. The kernel releases the lock automatically when the process exits, even on a crash, so there is nothing stale to clean up.
• Let systemd manage lifecycle. A systemd service tracks the process through its cgroup, so it always knows whether the daemon is alive without a PID file. Prefer Type=simple/Type=notify and avoid PIDFile= unless a forking daemon requires it.
• Write PID files atomically. If you must write one, write to a temporary file and rename(2) it into place so readers never see a partial PID.
• Verify liveness properly. A robust check does not trust the PID alone; it also confirms /proc/<pid>/comm or the exe symlink matches the expected program, defeating PID reuse.
• Reset failed units. After clearing a stale lock, run systemctl reset-failed <unit> so systemd forgets the previous failure and start rate limits do not block you.
• Store locks under /run. /run (and the /var/run symlink to it) is a tmpfs cleared on every boot, so lock files there cannot survive a reboot as stale artifacts.

Quick Command Reference

# Is the recorded PID actually alive?
cat /run/app/app.pid
kill -0 "$(cat /run/app/app.pid)" && echo alive || echo dead
ps -p "$(cat /run/app/app.pid)" -o pid,user,comm,args
ls -l /proc/"$(cat /run/app/app.pid)"/exe   # defeat PID reuse

# Who holds the lock?
sudo lslocks
sudo fuser -v /run/app/app.lock
flock -n /run/app/app.lock -c true   # succeeds only if lock is free

# How old is the lock file?
stat -c '%y %n' /run/app/app.lock
uptime -s

# Clear a CONFIRMED-stale lock and restart cleanly
sudo rm -f /run/app/app.pid /run/app/app.lock
sudo systemctl reset-failed app.service
sudo systemctl start app.service

# Prevent recurrence: kernel-managed single instance
flock -n /run/app/app.lock -c 'your-command --run'

Conclusion

Resource temporarily unavailable from flock and another instance is already running from a PID file are both symptoms of a lock outliving the process that created it. The decisive question is always the same: is the process behind this lock actually alive? Answer it with kill -0, ps, and a peek at /proc/<pid>/exe to defeat PID reuse, then confirm the lock is free with a non-blocking flock -n and an mtime older than the last boot. Only after proving staleness should you remove the file and systemctl reset-failed the unit. Better still, hand the whole problem to the kernel and systemd: flock(1) wrappers and cgroup-based lifecycle tracking release locks automatically on exit, so stale locks stop happening in the first place.