🚨 Kafka Error Guide: 'Too many open files' — Raise File Descriptor Limits

Overview

A Kafka broker keeps a file descriptor open for every log segment, index, and network socket it touches. When the process exceeds its per-process descriptor limit, the JVM throws Too many open files and the broker can no longer open new segments or accept connections:

ERROR Error while accepting connection (kafka.network.Acceptor)
java.io.IOException: Too many open files
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
        at kafka.network.Acceptor.accept(SocketServer.scala:717)

The same underlying EMFILE condition also surfaces when a broker tries to roll or open a log segment:

ERROR Error while writing to checkpoint file (kafka.server.epoch.LeaderEpochFileCache)
java.io.FileNotFoundException: /var/lib/kafka/data/orders-7/00000000000000012345.index (Too many open files)

Symptoms

• Brokers log java.io.IOException: Too many open files in the Acceptor or when opening .log/.index/.timeindex files.
• New producer and consumer connections are refused while existing ones keep working for a while.
• Under-replicated partitions climb because replica fetchers cannot open sockets to peers.
• The broker may stay up but stop serving new clients, or fail during log recovery on restart.
• ls -l /proc/<pid>/fd | wc -l sits at or near the process ulimit -n.

Common Root Causes

• Default OS limit too low. A stock nofile limit of 1024 is far below what a broker with thousands of partitions needs.
• Too many partitions/segments per broker. Each partition contributes several open files (segment + index files); large clusters need tens or hundreds of thousands of descriptors.
• systemd ignores limits.conf. Under systemd, LimitNOFILE= in the unit file governs the process, not /etc/security/limits.conf.
• Small segment.bytes / short segment.ms. Tiny segments multiply the file count per partition.
• Leaked client connections. Applications that open producers/consumers without closing them pile up broker-side sockets.
• Monitoring/JMX and inter-broker sockets counted against the same limit on busy clusters.

Diagnostic Workflow

First confirm which broker is affected and how many partitions it hosts using the standard tools:

kafka-topics.sh --bootstrap-server localhost:9092 --describe \
  | grep -c 'Partition:'

kafka-topics.sh --bootstrap-server localhost:9092 --describe \
  --under-replicated-partitions

Find the broker PID and its actual open-file usage versus its limit:

KPID=$(pgrep -f kafka.Kafka)
ls -l /proc/$KPID/fd | wc -l          # current open descriptors
cat /proc/$KPID/limits | grep 'open files'

Break down what those descriptors are — sockets versus data files:

sudo lsof -p $KPID | awk '{print $5}' | sort | uniq -c | sort -rn | head
sudo lsof -p $KPID | grep -c 'sock'
sudo lsof -p $KPID | grep -c '\.log$'

Check the effective systemd limit and the broker log for the error:

systemctl show kafka -p LimitNOFILE
grep -i 'too many open files' /var/lib/kafka/logs/server.log

Example Root Cause Analysis

A broker started refusing new connections and its under-replicated partition count jumped. kafka-topics.sh --describe showed the broker hosted about 4,200 partitions. ls -l /proc/$KPID/fd | wc -l reported roughly 65,500 descriptors while /proc/$KPID/limits showed a hard limit of 65,536 — the process was pinned at its ceiling.

The host’s /etc/security/limits.conf set nofile to 262144, but the broker ran under systemd, and systemctl show kafka -p LimitNOFILE returned LimitNOFILE=65536. systemd was capping the process regardless of limits.conf. Each partition contributed a segment plus two index files, and with small segment.bytes the file count per partition was high, so 4,200 partitions alone consumed the bulk of the descriptors before sockets were even counted.

The fix was to set LimitNOFILE=1048576 in the systemd unit, daemon-reload, and restart the broker; open files then leveled off well below the new limit and under-replicated partitions recovered. Reducing segment fragmentation (larger segment.bytes) trimmed the steady-state file count further.

Prevention Best Practices

• Set LimitNOFILE to a large value (for example 1048576) in the systemd unit — do not rely on /etc/security/limits.conf when systemd manages Kafka.
• Budget descriptors: roughly estimate open files as (partitions per broker x files per segment x active segments) plus expected connections, and keep the limit well above that.
• Cap partitions per broker during capacity planning; spread partitions across more brokers rather than overloading a few.
• Use reasonably large segment.bytes and sane segment.ms so partitions don’t fragment into many small files.
• Alert on open descriptors as a percentage of the limit (for example at 80%), not just on the crash.
• Ensure client applications close idle producers/consumers so broker sockets don’t leak.

Quick Command Reference

# Current vs. limit for the broker process
KPID=$(pgrep -f kafka.Kafka)
ls -l /proc/$KPID/fd | wc -l
cat /proc/$KPID/limits | grep 'open files'

# Effective systemd limit
systemctl show kafka -p LimitNOFILE

# What the descriptors are
sudo lsof -p $KPID | awk '{print $5}' | sort | uniq -c | sort -rn | head

# Partition / URP counts
kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions

# Raise the limit (systemd unit), then:
sudo systemctl daemon-reload && sudo systemctl restart kafka

Conclusion

Too many open files is an operating-system limit, not a Kafka bug: the broker simply ran out of file descriptors for its log segments and sockets. Diagnose it by comparing the process’s live descriptor count against its effective limit — remembering that systemd’s LimitNOFILE overrides limits.conf — then raise the limit and reduce partition/segment fragmentation so the broker has comfortable headroom. Alerting on descriptor usage keeps the fix ahead of the next outage.