🚨 Loki Error Guide: 'too many open files' — Raise the File-Descriptor Limit

Overview

Loki hits this error when the process has exhausted its file-descriptor budget. Every network connection, open chunk, and index file consumes a descriptor, and once the OS limit is reached the listener can no longer accept new connections:

accept tcp [::]:3100: accept4: too many open files

On the ingester or querier the same exhaustion shows up when opening files or dialing storage:

socket: too many open files

This is an operating-system limit, not a Loki bug. The default file-descriptor ulimit — often 1024 under systemd or a container runtime — is far too low for a busy Loki that holds many client connections plus open chunk and index files and connections to memcached and object storage. The fix is to raise the descriptor limit at whichever layer applies (systemd unit, container/pod, or PAM limits) and to reduce needless connection churn.

Symptoms

• Loki logs accept4: too many open files and stops accepting new HTTP/gRPC connections while the process stays up.
• Ingester or querier logs socket: too many open files when dialing storage or memcached.
• ls /proc/<pid>/fd | wc -l sits at or just under the process limit.
• Errors coincide with a spike in concurrent queries or client connections, then clear when load drops.
• A freshly deployed Loki works, then fails under production connection volume with no config change.

Common Root Causes

• OS file-descriptor ulimit too low — the default 1024 cannot cover connections plus open chunk/index files.
• systemd default LimitNOFILE — the unit inherits a small soft limit that caps the process regardless of /etc/security/limits.conf.
• High query concurrency — many simultaneous queries each open chunks, index files, and storage connections.
• Many storage/memcached connections — pools to object storage and cache multiply descriptors per instance.
• Connection churn — clients or proxies open and abandon connections faster than they close, piling up sockets in CLOSE_WAIT.
• Container runtime cap — the pod/container ulimit is lower than the host and silently limits the process.

How to diagnose

1. Read the process’s actual limit and current usage — compare the two to confirm exhaustion:

   pid=$(pgrep -f 'loki -config')
   cat /proc/$pid/limits | grep 'open files'
   ls /proc/$pid/fd | wc -l

2. Check the shell/service default so you know what Loki inherited:

   ulimit -n            # soft limit
   ulimit -Hn           # hard limit

3. See what is consuming descriptors — sockets versus files points at the cause:

   ls -l /proc/$pid/fd | grep -c 'socket:'   # connection-heavy
   ls -l /proc/$pid/fd | grep -c '/chunks/'  # storage/file-heavy

4. For Kubernetes, verify the pod’s effective limit rather than the node’s:

   kubectl exec deploy/loki -- sh -c 'cat /proc/1/limits | grep "open files"'

5. Confirm the intended limit is expressed in config before you roll it out:

   # systemd drop-in target (see fix below)
   # [Service]
   # LimitNOFILE=1048576

Fixes

Raise LimitNOFILE in a systemd drop-in for a package or bare-metal install, then reload and restart so Loki inherits the higher limit:

# /etc/systemd/system/loki.service.d/override.conf
[Service]
LimitNOFILE=1048576

systemctl daemon-reload
systemctl restart loki
# verify the process picked it up
cat /proc/$(pgrep -f 'loki -config')/limits | grep 'open files'

Set the container/pod ulimit so Kubernetes or the runtime does not cap Loki below the host. With Docker:

docker run --ulimit nofile=1048576:1048576 grafana/loki:latest

For Kubernetes, raise it on the node’s container runtime (containerd LimitNOFILE) or via a privileged init container, since core pod specs do not expose nofile directly.

Raise the PAM/system limits for interactive and non-systemd launches via /etc/security/limits.conf:

# /etc/security/limits.conf
loki  soft  nofile  1048576
loki  hard  nofile  1048576

Reduce needless connection churn so Loki holds fewer sockets at once — enable keepalive on the proxy in front of Loki and cap client connection rates:

# nginx in front of Loki: reuse upstream connections instead of churning
upstream loki { server loki:3100; keepalive 64; }
proxy_http_version 1.1;
proxy_set_header Connection "";

Monitor open descriptors so you raise the limit before it is hit again — alert on the process fd count approaching its ceiling:

# fraction of the fd limit in use per Loki process
process_open_fds / process_max_fds > 0.8

What to watch out for

• Editing /etc/security/limits.conf does not affect systemd services — for those you must set LimitNOFILE in the unit or a drop-in, or the process keeps the old cap.
• A pod can be limited below its node without warning; always check /proc/1/limits inside the container, not the host.
• Raising the limit hides but does not fix a leak — if fd usage climbs steadily under steady load, look for connection churn or unclosed sockets, not just a bigger ceiling.
• The soft limit is what applies at runtime; raising only the hard limit leaves Loki capped until the soft limit is raised too.
• Do not pair a large LimitNOFILE with MemoryDenyWriteExecute=yes on the same unit — the latter breaks Go’s runtime and produces a different failure that can be mistaken for this one.

Related

• Loki Error Guide: ‘failed to flush chunks’ — the ingester write failure that fd exhaustion on storage connections can trigger.
• Loki Error Guide: ‘too many outstanding requests’ — read-path overload that often accompanies a connection-descriptor spike.
• Loki Error Guide: ‘context deadline exceeded’ — the timeout that surfaces when a fd-starved listener cannot accept connections in time.