🚨 OpenTelemetry Error Guide: 'too many open files' in the Collector — Fix File Descriptor Limits

Overview

This error appears when the Collector process has exhausted its file-descriptor limit and the kernel refuses to hand out another one. Because every accepted network connection and every open log file consumes an fd, a busy Collector can hit the ceiling and start failing to accept new OTLP connections:

2026-07-12T14:22:10.114Z	error	otlpreceiver/otlp.go:158	Failed to accept connection	{"kind": "receiver", "name": "otlp", "error": "accept tcp [::]:4317: accept4: too many open files"}

The filelog receiver surfaces the same underlying EMFILE when it tries to open matched files:

error	fileconsumer/file.go:132	Failed to open file	{"kind": "receiver", "name": "filelog", "error": "open /var/log/pods/app-0/app/0.log: too many open files"}

too many open files (EMFILE) means the process reached its per-process open-fd limit (RLIMIT_NOFILE). New connections are rejected and log files go unread until fds are freed or the limit is raised.

Symptoms

• The Collector logs too many open files / accept4: too many open files under load.
• New OTLP clients get connection refused or timeouts because the listener can’t accept.
• The filelog receiver stops tailing new files and telemetry gaps appear for busy pods.
• Errors scale with connection count or the number of matched log files, not with payload size.
• ls /proc/<pid>/fd | wc -l sits at or near the process’s Max open files limit.
• Restarting the Collector clears it temporarily, then it recurs as connections/files accumulate.

Common Root Causes

• Default systemd fd limit too low — the unit runs with the distro default LimitNOFILE (often 1024), far below what a busy gateway needs.
• High concurrent gRPC connection count — thousands of SDK clients each hold an OTLP connection, each consuming an fd on the receiver.
• filelog watching many files — a broad include glob matches thousands of container logs, each an open fd.
• Leaked connections — clients that never close, or a missing max_connection_age, so fds accumulate over time.
• Container/pod limit not propagated — the host is generous but the container’s ulimit or the pod’s limits are not.
• max_open_files unset on filelog — the receiver keeps too many log files open simultaneously instead of rotating through them.

Diagnostic Workflow

Find the Collector PID and compare its current open fds against its limit — if they are close, the limit is the problem:

pid=$(pgrep -f otelcol-contrib)
# Current open fds vs the hard/soft limit
ls /proc/"$pid"/fd | wc -l
grep 'Max open files' /proc/"$pid"/limits

For a systemd-managed Collector, raise LimitNOFILE in a drop-in so the limit survives restarts and upgrades:

# /etc/systemd/system/otelcol-contrib.service.d/limits.conf
mkdir -p /etc/systemd/system/otelcol-contrib.service.d

[Service]
LimitNOFILE=262144

systemctl daemon-reload
systemctl restart otelcol-contrib
grep 'Max open files' /proc/"$(pgrep -f otelcol-contrib)"/limits

If the filelog receiver is the consumer, cap how many files it holds open at once and bound the receiver so a broad glob can’t exhaust fds:

receivers:
  filelog:
    include:
      - /var/log/pods/*/*/*.log
    max_concurrent_files: 1024   # cap simultaneously open files
    start_at: end

  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        keepalive:
          server_parameters:
            max_connection_age: 30s        # recycle connections so fds are freed
            max_connection_age_grace: 5s

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [batch]
      exporters: [otlp]

Validate and confirm the new limit took effect, and watch for recurrence:

otelcol-contrib validate --config /etc/otelcol-contrib/config.yaml
journalctl -u otelcol-contrib -f | grep -i 'too many open files\|accept4'

Example Root Cause Analysis

A gateway Collector installed from the vendor RPM ran under systemd with the default LimitNOFILE=1024. As the platform grew to ~1,800 application pods, each holding a persistent gRPC connection, the receiver crossed 1,024 open fds during the morning ramp and began logging accept tcp [::]:4317: accept4: too many open files. New pods could not connect, and their traces were silently lost while existing connections kept working.

The fix had two parts. First, a systemd drop-in set LimitNOFILE=262144, giving the process ample headroom for the connection count plus filelog fds. Second, max_connection_age: 30s was added to the gRPC receiver so long-lived connections were periodically recycled — freeing fds and rebalancing clients across gateway replicas instead of pinning them to one. After daemon-reload and restart, /proc/<pid>/limits showed the raised ceiling, open fds settled around 4,000, and the accept errors stopped entirely.

Prevention Best Practices

• Ship a systemd drop-in with a generous LimitNOFILE (e.g. 262144) as part of the Collector’s standard install, never the distro default.
• Set max_connection_age on the gRPC receiver so connections recycle and fds are released instead of accumulating.
• Bound the filelog receiver with max_concurrent_files and a targeted include glob rather than a wildcard over every log.
• Alert on /proc/<pid>/fd count relative to the limit so you see saturation before connections are refused.
• Propagate limits into containers (--ulimit nofile= / pod securityContext), not just the host.
• Scale gateway Collectors horizontally so no single instance carries every connection and log file.

Quick Command Reference

# Current open fds for the Collector
pid=$(pgrep -f otelcol-contrib); ls /proc/"$pid"/fd | wc -l

# The process's open-file limit
grep 'Max open files' /proc/"$pid"/limits

# Apply a raised limit via systemd drop-in
systemctl daemon-reload && systemctl restart otelcol-contrib

# Watch for recurrence live
journalctl -u otelcol-contrib -f | grep -i 'too many open files'

Conclusion

too many open files means the Collector process hit its RLIMIT_NOFILE ceiling — every accepted connection and every tailed log file costs an fd, and a busy gateway blows past the default 1024 fast. Raise LimitNOFILE with a systemd drop-in, recycle connections with max_connection_age, and bound the filelog receiver with max_concurrent_files. Then alert on the fd count versus the limit so you scale ahead of exhaustion instead of discovering it as refused connections.

Related

• OpenTelemetry Error Guide: ‘address already in use’
• OpenTelemetry Error Guide: ‘connection refused’ to the Collector
• OpenTelemetry Error Guide: ‘sending queue is full’