🚨 VictoriaMetrics Error: 'cannot open file ... no such file or directory' — Cause, Fix, and Troubleshooting Guide

Overview

VictoriaMetrics stores each ingested block as an immutable “part” under -storageDataPath. On startup and during background merges, vmstorage (or single-node victoria-metrics) opens the index and data files that make up every part. When one of those files is missing, the process logs a fatal error and refuses to serve the affected data:

cannot open file "/victoria-metrics-data/data/small/2026_07/0000..._index.bin": open ...: no such file or directory

This almost always means a part was left half-written — the on-disk directory exists but one of its files never made it to disk. The most common trigger is a process killed with SIGKILL (an OOM kill or kill -9) mid-flush, but it can also come from filesystem corruption or a disappearing storage volume. VictoriaMetrics is designed to self-heal from partial parts on a clean restart, so the fix is usually to let it clean up and then address the root cause.

Symptoms

• vmstorage or single-node victoria-metrics exits immediately on startup with cannot open file ... no such file or directory.
• The path in the error points inside -storageDataPath (for example under data/small/<partition>/ or data/big/).
• Recent data is unqueryable, or the whole node stays down and never reaches a ready state.
• The crash was preceded by an OOM kill, a hard reboot, a full disk, or an unmount of the storage volume.
• The failing part directory exists but is missing index.bin, values.bin, timestamps.bin, or metaindex.bin.

Common Root Causes

• Process killed with SIGKILL / OOM — the kernel OOM-killer or a kill -9 stopped VM mid-flush, leaving a part directory without all of its files.
• Disk or filesystem corruption — bad sectors or an fsck that quarantined files removed data VM expected to find.
• Flaky network storage — -storageDataPath on NFS or a network block device that briefly disappeared or returned partial writes.
• Manual deletion of data files — someone removed files under the data directory by hand, or a cleanup script matched them.
• Disk filled then truncated — the volume ran out of space during a flush, so a file was created but never fully written.

How to diagnose

First, read the full startup log around the error. VictoriaMetrics reports when it discovers and removes incomplete parts, and those lines tell you whether it is self-healing:

journalctl -u vmstorage --no-pager | grep -Ei 'incomplete part|cannot open file|removing'

Look for a graceful cleanup line such as deleting incomplete part — if VM logged that and then started, the part was partial and has been discarded. Inspect the referenced directory to confirm which file is missing:

# Sanitized example path from the error
ls -la /victoria-metrics-data/data/small/2026_07/0000..._<part>/

Rule out hardware and filesystem faults before touching data:

# Kernel messages: OOM kills, I/O errors, filesystem read-only events
dmesg -T | grep -Ei 'oom|i/o error|ext4|xfs|remount'

# SMART health for the underlying disk
smartctl -H /dev/sdX

Confirm the storage volume is actually mounted and writable:

mount | grep victoria-metrics-data
touch /victoria-metrics-data/.wtest && rm /victoria-metrics-data/.wtest && echo "writable"

Fixes

1. Restart and let VictoriaMetrics remove the incomplete part. On a clean start VM detects a partial part, logs deleting incomplete part, and drops it automatically. Stop the process (gracefully, see fix 5) and start it again:

systemctl restart vmstorage
journalctl -u vmstorage -f | grep -i 'incomplete part'

You lose only the unflushed block that was never fully persisted; already-merged data is intact.

2. Check the disk and filesystem for real corruption. If dmesg shows I/O errors or SMART reports failures, the missing file is a symptom, not the cause. Move -storageDataPath to healthy storage and run an fsck on the affected volume while VM is stopped.

3. Verify the storage mount is present and healthy. Ensure -storageDataPath points at a mounted, writable volume before start. A path that resolves to an empty local directory because the real volume failed to mount will produce this error for every part.

4. Restore from a vmbackup snapshot. If a merged part (not just an in-flight one) is genuinely gone, restore the data directory from your last snapshot:

./vmrestore -src=s3://vm-backups/daily/ -storageDataPath=/victoria-metrics-data

Start vmstorage only after the restore completes.

5. Always stop VictoriaMetrics gracefully. Send SIGINT (or systemctl stop, which does this) so VM flushes in-memory parts and closes files cleanly. Never use SIGKILL / kill -9 except as a last resort, and give the service a generous TimeoutStopSec so systemd does not escalate to SIGKILL during a long flush.

What to watch out for

• A single restart usually resolves partial-part errors; if the same file error returns after every restart, suspect the underlying disk or a read-only filesystem, not VM.
• Do not delete whole partition directories to “fix” the error — you can discard good data. Only VM’s own deleting incomplete part cleanup is safe.
• Set memory limits (-memory.allowedPercent) so the OOM-killer is less likely to SIGKILL VM mid-flush in the first place.
• Network-backed -storageDataPath (NFS, remote block storage) is a frequent source of partial writes; local SSD is strongly preferred for the data directory.
• Keep regular vmbackup snapshots so restore is an option when a merged part is lost rather than just an incomplete one.

Related

• VictoriaMetrics Error Guide: cannot obtain lock
• VictoriaMetrics Error Guide: no space left on device
• VictoriaMetrics Error Guide: read-only mode