🚨 Filebeat Error: 'too many open files' — Cause, Fix, and Troubleshooting Guide

Overview

Filebeat holds a file descriptor for every harvested file, plus its registry and output sockets. When the process hits its RLIMIT_NOFILE ceiling, the kernel returns EMFILE and any further open() or socket() fails:

Harvester could not be started on new file: /var/log/app/07.log, Err: error setting up harvester: Failed opening /var/log/app/07.log: open /var/log/app/07.log: too many open files

The same limit can also break the Elasticsearch/Logstash connection (dial tcp: socket: too many open files) because every open harvester and every output worker consumes a descriptor. The default limit for a service is often 1024, which a broad glob exhausts quickly. Filebeat keeps running but stops harvesting new files and may fail to reconnect its output.

Symptoms

• too many open files on harvester start or on output dial/socket.
• Newest log files stop being shipped while older, already-open ones continue.
• /proc/<pid>/fd count sits at the nofile soft limit.
• The error scales with the number of matched files — worse after adding paths or during log bursts.

Common Root Causes

• Low LimitNOFILE — the systemd unit inherits the default 1024 soft limit.
• Broad globs — **/*.log matching thousands of files, each held open.
• Harvesters never closing — close.* timers too long, so descriptors accumulate.
• Leaked descriptors on rotation when clean_* options are misconfigured.

How to diagnose

Inspect the effective limit and live descriptor usage:

cat /proc/$(pgrep -x filebeat)/limits | grep 'open files'
ls /proc/$(pgrep -x filebeat)/fd | wc -l

See what the descriptors are pointing at (files vs sockets):

sudo ls -l /proc/$(pgrep -x filebeat)/fd | awk '{print $NF}' | \
  sed 's/[0-9]*$//' | sort | uniq -c | sort -rn | head

Fixes

Raise the limit in the systemd unit — this is the durable fix:

# /etc/systemd/system/filebeat.service.d/override.conf
[Service]
LimitNOFILE=65536

systemctl daemon-reload
systemctl restart filebeat
cat /proc/$(pgrep -x filebeat)/limits | grep 'open files'   # verify 65536

Reduce demand for descriptors by capping harvesters and closing idle files:

filebeat.inputs:
  - type: filestream
    id: app
    paths:
      - /var/log/app/*.log
    harvester_limit: 512
    close.on_state_change.inactive: 5m
    close.reader.after_interval: 5m

Tighten globs so Filebeat is not holding thousands of files open, then restart and confirm the fd count stays well under the new limit.

What to watch out for

• Editing /etc/security/limits.conf does not affect systemd services — use a unit override.
• The output socket needs descriptors too; leaving no headroom breaks reconnects, not just harvesting.
• harvester_limit throttles concurrency but a very broad glob can still queue endlessly — scope paths.
• Verify the new limit from /proc/<pid>/limits after restart; a typo silently keeps the old value.

Related

• Filebeat Error: ‘Harvester crawler could not be started’
• Filebeat Error Guide: ‘Harvester could not be started … permission denied’
• Filebeat Error Guide: ‘connection refused’