🚨 VictoriaMetrics Error Guide: 'too many open files' — Raise File Descriptor Limits

Overview

VictoriaMetrics keeps many data part files open (and memory-maps index files) for fast reads. When the process hits the operating system’s per-process file-descriptor cap, any operation that needs a new descriptor fails with:

2026-07-09T10:14:22.031Z error VictoriaMetrics/lib/mergeset/table.go:... : cannot open file "/victoria-metrics-data/data/small/.../part.bin": open ...: too many open files

The same too many open files (EMFILE) surfaces on socket accepts and on opening new parts during background merges. It is almost never a VictoriaMetrics bug — it is an under-provisioned nofile limit for the workload’s cardinality and part count.

Symptoms

• Logs contain too many open files on open, accept, or mmap operations.
• New scrapes/queries fail while the process stays running.
• Background merges stall; vm_pending_rows climbs.
• process_open_fds approaches process_max_fds in the metrics endpoint.
• Errors appear after data or cardinality growth, or after switching off mmap.

Common Root Causes

• Low LimitNOFILE — the systemd unit or container runtime leaves the default (often 1024), far too low for VictoriaMetrics.
• High cardinality / many parts — large numbers of small parts (from high churn) each consume descriptors until merged.
• mmap disabled — running with -fs.disableMmap forces regular file handles, multiplying descriptor use.
• Many concurrent connections — a large client fleet consuming sockets alongside data-file descriptors.
• Container defaults — Kubernetes/Docker not raising nofile, so the pod inherits a tiny limit.
• Slow merges — undersized CPU/disk leaves many small parts open longer than normal.

Diagnostic Workflow

Compare current usage against the ceiling directly from the metrics endpoint (port 8428, or 8482 for vmstorage):

curl -s http://localhost:8428/metrics | grep -E 'process_(open_fds|max_fds)'

If process_open_fds is near process_max_fds, the limit is the problem. Check the effective limit on the running process:

VMPID=$(pgrep -f victoria-metrics)
cat /proc/$VMPID/limits | grep 'open files'
ls /proc/$VMPID/fd | wc -l

For a systemd-managed instance, read the configured limit:

systemctl show victoriametrics -p LimitNOFILE

In Kubernetes, confirm the container’s limit:

kubectl exec vmstorage-0 -- sh -c 'cat /proc/1/limits | grep "open files"'

Look at part counts and merge backlog in vmui with MetricsQL — a high part count means more open descriptors:

vm_parts{type="storage/small"}
vm_pending_rows

Example Root Cause Analysis

A single-node VictoriaMetrics started logging too many open files after ingestion doubled. process_open_fds sat at 1024 against process_max_fds of 1024, and /proc/$PID/limits confirmed a soft limit of 1024 — the binary had been launched from a shell without a raised ulimit -n. High churn from a noisy Kubernetes label had also produced thousands of small parts (vm_parts{type="storage/small"} was very high), so many descriptors were held open awaiting merges.

The fix had two parts. First, a systemd drop-in set LimitNOFILE=1048576, immediately clearing the errors. Second, metric_relabel_configs on vmagent dropped the churning label, cutting small-part count by ~80% so descriptor pressure stayed low even under load.

Prevention Best Practices

• Set LimitNOFILE high (VictoriaMetrics docs recommend a large value such as 1048576) in the systemd unit or container spec.
• In Kubernetes, raise the pod’s file limit via the runtime or an init container; do not rely on the default.
• Keep mmap enabled (the default) unless a specific filesystem forces -fs.disableMmap; mmap uses fewer descriptors.
• Control cardinality and churn so the number of open part files stays bounded.
• Provision enough CPU/disk for merges to keep small-part counts low.
• Alert on process_open_fds / process_max_fds > 0.8 to act before exhaustion.

Quick Command Reference

# Open vs max descriptors
curl -s http://localhost:8428/metrics | grep -E 'process_(open_fds|max_fds)'

# Effective limit on the running process
cat /proc/$(pgrep -f victoria-metrics)/limits | grep 'open files'

# systemd drop-in to raise the limit
# /etc/systemd/system/victoriametrics.service.d/limits.conf
# [Service]
# LimitNOFILE=1048576
sudo systemctl daemon-reload && sudo systemctl restart victoriametrics

# Part count (run in vmui)
# vm_parts{type="storage/small"}

Conclusion

too many open files is an OS resource limit, not a VictoriaMetrics defect. Confirm it by comparing process_open_fds with process_max_fds and reading /proc/$PID/limits, then raise LimitNOFILE to a large value in your systemd unit or container spec. Pair that with cardinality and churn control so the number of open part files stays bounded — together they keep VictoriaMetrics comfortably under any reasonable descriptor ceiling.