🚨 Linux Error: 'Failed to allocate directory watch: Too many open files' — Cause, Fix, and Troubleshooting Guide

Overview

Despite the “Too many open files” tail, this error is almost never about ordinary file-descriptor limits. It is inotify watch exhaustion: a process asked the kernel to watch a directory and the per-user inotify budget was already spent.

Failed to allocate directory watch: Too many open files

inotify is the kernel subsystem that notifies programs when files change. It has two per-user ceilings — fs.inotify.max_user_watches (how many paths one user can watch in total) and fs.inotify.max_user_instances (how many separate inotify handles one user can open). When either runs out, inotify_add_watch() / inotify_init() returns ENOSPC or EMFILE, and callers such as systemd, journald, udev, file-sync agents, IDE language servers, and Kubernetes kubelets print this exact line. It is distinct from a process hitting its ulimit -n — see the companion guide on Too many open files for the plain descriptor case. This guide is about the watch/instance ceilings and how to tell them apart.

Symptoms

• systemd or a .path unit logs Failed to allocate directory watch: Too many open files and stops reacting to file changes.
• File watchers appear “stuck”: an IDE, tail -f-style log shipper, or inotifywait silently misses events or exits with “No space left on device” (ENOSPC) from inotify_add_watch.
• On Kubernetes nodes, kubelet or a CNI/CSI agent logs inotify allocation failures once many pods and ConfigMap mounts are watched.
• systemctl status for a .path unit shows it failing to arm, even though descriptors (/proc/<pid>/fd) are nowhere near the process limit.

journalctl -xe --no-pager | grep -i 'directory watch'

Jul 12 10:41:22 node-03 systemd[1]: Failed to allocate directory watch: Too many open files
Jul 12 10:41:22 node-03 filesync[2210]: inotify_add_watch(/srv/data) failed: No space left on device

Common Root Causes

1. fs.inotify.max_user_watches is too low for the workload

The historical default (often 8192, sometimes 65536 on newer distros) is trivially exceeded by editors that watch an entire node_modules, log shippers tailing thousands of paths, backup/file-sync agents, or a node running dozens of pods with mounted ConfigMaps and Secrets.

2. fs.inotify.max_user_instances is exhausted

Every process that calls inotify_init() consumes one instance. Many small watchers under the same UID (containers all running as root, several agents, per-tab IDE processes) can hit the instance ceiling long before the watch ceiling.

3. Many services or containers share one UID’s budget

The ceilings are per user (UID), not per process. A host where dozens of containers or daemons all run as UID 0 pool their watches into one budget, so no single process looks abusive while the shared total is full.

4. A watcher leaks watches

An application that adds watches on rotated/recreated files without removing the stale ones climbs steadily until ENOSPC, independent of load — a leak, not a small ceiling.

5. It is actually FD exhaustion, not watches

Occasionally the same “Too many open files” string comes from inotify_init() returning EMFILE because the process is out of file descriptors — a genuine ulimit -n problem wearing an inotify costume. Distinguishing the two is step one.

How to diagnose

Step 1: Read the current inotify ceilings

sysctl fs.inotify.max_user_watches fs.inotify.max_user_instances

fs.inotify.max_user_watches = 8192
fs.inotify.max_user_instances = 128

A watch ceiling in the low thousands on a busy node or dev box is the usual culprit.

Step 2: Decide whether it is a watch or an FD problem

# The failing process
PID=$(pgrep -o filesync)

# Watches this process holds (count of inotify anon inodes across its fds)
find /proc/$PID/fd -lname 'anon_inode:inotify' 2>/dev/null | wc -l

# Descriptors vs the process limit (the ulimit -n / EMFILE angle)
ls /proc/$PID/fd | wc -l
grep 'open files' /proc/$PID/limits

If the fd count is far below the “open files” soft limit but events are failing, it is a watch/instance ceiling — this guide. If the fd count is pinned at the soft limit, it is plain descriptor exhaustion — raise LimitNOFILE/ulimit -n and see the Too many open files guide instead.

Step 3: Find who is spending the per-UID budget

# Approximate watch consumers: sum inotify instances per PID
for p in /proc/[0-9]*; do
  c=$(find "$p/fd" -lname 'anon_inode:inotify' 2>/dev/null | wc -l)
  [ "$c" -gt 0 ] && echo "$c $(cat "$p/comm" 2>/dev/null) ${p##*/}"
done | sort -rn | head

Remember the budget is per UID, so add up the offenders sharing a user, not just the single largest process.

Step 4: Watch it fill under real load

watch -n5 'sysctl -n fs.inotify.max_user_watches; \
  find /proc/*/fd -lname "anon_inode:inotify" 2>/dev/null | wc -l'

A count that climbs and never falls under steady conditions is a leak; a count that sits just under the ceiling under normal load is simply a low ceiling.

Fixes

Raise the watch and instance ceilings live, then persist them

# Immediate (until reboot)
sudo sysctl -w fs.inotify.max_user_watches=524288
sudo sysctl -w fs.inotify.max_user_instances=1024

# Persist via /etc/sysctl.d/ so it survives reboot
sudo tee /etc/sysctl.d/60-inotify.conf >/dev/null <<'EOF'
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 1024
EOF

# Apply all sysctl.d drop-ins
sudo sysctl --system

Verify:

sysctl fs.inotify.max_user_watches fs.inotify.max_user_instances

524288 watches is a common, safe value for dev boxes and busy nodes; each watch costs roughly 1 KB of unswappable kernel memory, so budget accordingly (524288 watches ≈ 512 MB worst case if fully used).

If it is actually FD exhaustion, raise LimitNOFILE instead

When step 2 showed the fd count pinned at the soft limit, the fix is the descriptor limit, not inotify:

# For a systemd service
sudo systemctl edit filesync.service     # add:  [Service]\nLimitNOFILE=65535
sudo systemctl daemon-reload
sudo systemctl restart filesync.service

# Confirm
grep 'open files' /proc/$(pgrep -o filesync)/limits

Restart the consumer so it re-arms its watches

After raising ceilings, the watcher must re-request its watches:

sudo systemctl restart filesync.service
journalctl -u filesync.service -xe --no-pager | tail -5

If it is a leak, fix the code

Raising ceilings only delays a genuine watch leak. Ensure the application removes watches (inotify_rm_watch) on deleted/rotated files instead of accumulating them.

What to watch out for

• This is not ulimit -n. Raising LimitNOFILE does nothing for watch exhaustion, and raising max_user_watches does nothing for real FD exhaustion. Run the step-2 check before changing anything.
• The ceiling is per UID, not per process. Consolidating everything under one user (many root containers) makes the shared budget the bottleneck; the largest single process may look innocent.
• Watches cost unswappable kernel memory. Don’t set max_user_watches to an enormous number “to be safe” on a small node — size it to the workload.
• /etc/sysctl.conf edits need sysctl --system (or reboot) to take effect; a value set only with sysctl -w reverts on reboot.
• ENOSPC from inotify_add_watch is the watch ceiling; EMFILE from inotify_init is the instance/FD ceiling. The errno tells you which knob to turn.
• Container images inherit the host’s sysctls. You cannot fix node-level inotify limits from inside an unprivileged container — set them on the node.

Related

• Linux Error: Too many open files
• Linux Error Guide: ‘Failed to start’ systemd Service Won’t Start
• Linux Error: fork: Resource temporarily unavailable
• All Linux admin guides

Want faster Linux incident response? Use the free incident assistant to turn inotify and descriptor errors into clear diagnostics and reusable runbooks.