🚨 Kubernetes Error Guide: 'too many open files' — Fix inotify and File Descriptor Limits

Overview

too many open files is a kernel resource-exhaustion error (errno EMFILE/ENFILE) surfacing inside a pod, in the kubelet, or in a controller. A process tried to open a file, socket, or inotify watch and the kernel refused because a limit was reached. In Kubernetes this shows up in two very different flavors that people constantly conflate:

• Per-process file descriptors — the classic RLIMIT_NOFILE ulimit; a leaking app or a legitimately busy proxy runs out of fds.
• inotify watches/instances — the node-wide fs.inotify.max_user_watches / max_user_instances sysctls, exhausted by controllers, log shippers, config-reloaders, and file-watching frameworks across all pods on the node.

A typical log line:

2026-07-06T10:14:02Z ERROR failed to watch /etc/config: too many open files

or, more alarmingly, from the node’s own components:

kubelet: failed to create inotify: too many open files
E0706 reflector.go:138 watch of *v1.ConfigMap ended with: too many open files

The inotify variant is a node-level problem: one noisy pod can exhaust the shared watch table and take down the kubelet and every file-watcher on that node.

Symptoms

• App logs contain too many open files, EMFILE, or accept: too many open files.
• Config-reload / hot-reload stops working; a sidecar logs failed to create inotify.
• The kubelet or a controller logs watch ... ended with: too many open files and the node goes NotReady or flaps.
• dmesg shows VFS: file-max limit reached (system-wide fd exhaustion).

kubectl logs api-7c9d8f6b4d-2xqlk | grep -i 'too many open files' | tail -3

2026-07-06T10:14:02Z ERROR accept tcp [::]:8080: accept4: too many open files

kubectl describe node worker-2 | grep -i 'NotReady\|inotify'

Common Root Causes

1. App leaks file descriptors (unclosed files or sockets)

The most common in-pod cause: the app opens connections or files and never closes them, climbing until it hits its fd limit. Restarting masks it; the fd count creeping up over time confirms it.

# Find the app PID inside the container and count its open fds
kubectl exec api-7c9d8f6b4d-2xqlk -- sh -c 'ls /proc/1/fd | wc -l'
kubectl exec api-7c9d8f6b4d-2xqlk -- sh -c 'cat /proc/1/limits | grep "open files"'

1021
Max open files            1024                 1024                 files

1021/1024 and climbing is a leak (or an under-sized limit for a busy service). Fix the leak, or raise the limit if the load is legitimate.

2. inotify watches exhausted node-wide

Config-reloaders, log agents (Fluent Bit, Promtail), and frameworks that watch files consume inotify watches. Summed across all pods, they blow past the node’s fs.inotify.max_user_watches, and new watchers (including the kubelet) fail.

# On the node
sysctl fs.inotify.max_user_watches fs.inotify.max_user_instances

fs.inotify.max_user_watches = 8192
fs.inotify.max_user_instances = 128

An 8192 watch limit is tiny for a busy node. Raising these sysctls (via a DaemonSet or node config) is the usual fix.

3. Under-sized RLIMIT_NOFILE for a high-connection workload

A reverse proxy, API gateway, or database legitimately needs tens of thousands of fds. The default ulimit (often 1024) is simply too low, so it hits the wall under normal load, not because of a leak.

kubectl exec gateway-0 -- sh -c 'cat /proc/1/limits | grep "open files"'

4. System-wide file-max reached on the node

Rare, but severe: the sum of all open files on the host exceeds fs.file-max. This affects everything on the node, not one pod.

# On the node
cat /proc/sys/fs/file-nr        # allocated, unused, max
sysctl fs.file-max

1600000 0 1610000

allocated near max means the whole node is out of fds.

5. Kubelet / controller-runtime watch storms

Operators and controllers open API watch connections and inotify watches. A buggy operator that opens watches without bounds, or many operators on one node, can exhaust inotify instances and break the kubelet’s own watches.

Diagnostic Workflow

Step 1: Determine which flavor you have

kubectl logs <POD> | grep -i 'too many open files' | tail -5

accept:/socket messages point at per-process fds; failed to create inotify/watch ... ended points at inotify sysctls. This split decides everything downstream.

Step 2: For fd exhaustion — measure usage vs. limit

kubectl exec <POD> -- sh -c 'ls /proc/1/fd | wc -l'
kubectl exec <POD> -- sh -c 'cat /proc/1/limits | grep "open files"'

Usage pinned at the limit = leak or under-sized ulimit. Sample twice a minute apart to see if it climbs (leak) or is steady-at-ceiling (under-sized).

Step 3: For inotify — check the node sysctls and consumers

# On the node
sysctl fs.inotify.max_user_watches fs.inotify.max_user_instances
# Which processes hold inotify instances
find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null | cut -d/ -f3 | sort | uniq -c | sort -rn | head

The top consumers reveal whether a log agent, reloader, or operator is the culprit.

Step 4: Check node-wide fd allocation

# On the node
cat /proc/sys/fs/file-nr
sysctl fs.file-max

Rules out (or confirms) a host-level file-max ceiling.

Step 5: Check kubelet health

kubectl get nodes
journalctl -u kubelet --no-pager | grep -i 'inotify\|too many open files' | tail

If the kubelet itself logs the error, treat it as a node incident, not an app bug.

Example Root Cause Analysis

A config-reloader sidecar across many pods stops picking up ConfigMap changes, and worker-4 starts flapping NotReady.

kubectl logs reloader-abc -c reloader | tail -2

2026-07-06T10:14:02Z ERROR failed to create inotify watcher: too many open files

That is the inotify flavor. Check the node:

sysctl fs.inotify.max_user_instances

fs.inotify.max_user_instances = 128

Count instances by process:

find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null | cut -d/ -f3 | sort | uniq -c | sort -rn | head -3

  61 4172
  40 4188
  33 4201

Three heavy watchers plus the kubelet’s own watches exceed the 128-instance cap, so new watchers — including kubelet reflectors — fail, and the node flaps. Fix by raising the node sysctls (rolled out via a privileged DaemonSet or node bootstrap config):

# Applied on the node (persist in /etc/sysctl.d/ for reboots)
sysctl -w fs.inotify.max_user_instances=1024
sysctl -w fs.inotify.max_user_watches=524288

The kubelet’s watches recover, the reloader re-establishes its inotify watcher, and worker-4 returns to Ready. The durable fix is to bake generous inotify sysctls into the node image and to alert on inotify instance usage per node.

Prevention Best Practices

• Raise fs.inotify.max_user_watches and fs.inotify.max_user_instances in the node image/bootstrap to values sized for your densest node — the defaults are far too low for busy clusters.
• Set an appropriate RLIMIT_NOFILE for high-connection workloads (proxies, gateways, databases) rather than relying on the 1024 default.
• Treat a steadily climbing fd count as a leak and fix the app; do not paper over it by only raising the limit.
• Monitor inotify instance/watch usage and node-level file-nr so you catch exhaustion before the kubelet is affected.
• Audit config-reloaders, log shippers, and operators for how many watches they open per pod; consolidate where possible.
• Bake sysctl changes into node provisioning (or a privileged DaemonSet) so they survive reboots and autoscaler-created nodes. See more in Kubernetes & Helm guides.

Quick Command Reference

# Which flavor? (fd vs inotify)
kubectl logs <POD> | grep -i 'too many open files' | tail -5

# Per-process fd usage vs limit
kubectl exec <POD> -- sh -c 'ls /proc/1/fd | wc -l'
kubectl exec <POD> -- sh -c 'cat /proc/1/limits | grep "open files"'

# Node inotify limits and top consumers
sysctl fs.inotify.max_user_watches fs.inotify.max_user_instances
find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null | cut -d/ -f3 | sort | uniq -c | sort -rn | head

# Node-wide fd allocation
cat /proc/sys/fs/file-nr
sysctl fs.file-max

# Kubelet impact
journalctl -u kubelet --no-pager | grep -i 'inotify\|too many open files' | tail

# Raise inotify limits on the node (persist in /etc/sysctl.d/)
sysctl -w fs.inotify.max_user_instances=1024
sysctl -w fs.inotify.max_user_watches=524288

Conclusion

too many open files is kernel resource exhaustion with two distinct flavors that demand different fixes. The usual root causes:

1. An app leaks file descriptors (unclosed files/sockets) until it hits its ulimit.
2. inotify watches/instances are exhausted node-wide by reloaders, log agents, and operators.
3. RLIMIT_NOFILE is simply too low for a high-connection workload.
4. The node’s system-wide fs.file-max is reached, affecting everything on the host.
5. A watch storm from controllers breaks the kubelet’s own watches and flaps the node.

First decide whether the message is about per-process fds or inotify — the log wording tells you — then measure usage against the relevant limit. inotify exhaustion is a node incident because it can take down the kubelet; size those sysctls generously in the node image. For ad-hoc triage, the free incident assistant can turn a node or pod log dump into the likely root cause.