🚨 VictoriaMetrics Error Guide: 'cannot obtain lock on file flock.lock' — Ensure a Single Owner of the Data Directory

Overview

VictoriaMetrics takes an exclusive advisory lock on its data directory at startup so that only one process can ever write to it. When a second instance tries to open the same directory, or when a previous process is still holding the lock, startup aborts with:

cannot obtain lock on file "/victoria-metrics-data/flock.lock": another instance of VictoriaMetrics is already running or the lock file is stale

On the cluster vmstorage component the same guard protects -storageDataPath:

FATAL: cannot obtain lock on file "/vmstorage-data/flock.lock": resource temporarily unavailable

This is a safety feature: two writers on one data directory would corrupt the on-disk parts. The error means exactly one of two things — a second VictoriaMetrics process really is running against that directory, or a previous process died without releasing the lock and (rarely) something is holding a stale reference. The fix is to guarantee a single owner of the data path, then start cleanly.

Symptoms

• VictoriaMetrics (or vmstorage) exits immediately at boot with a FATAL lock message and never binds its port.
• systemctl start victoriametrics reports the service failed within a second or two.
• 8428 (single-node) or 8482/8401/8400 (vmstorage) never come up; scrapers and vmselect report the node as down.
• The error mentions flock.lock inside the configured -storageDataPath / -storageDataPath directory.
• It appears right after a container restart, a config change that spawned a second instance, or a host that was hard-rebooted.
• Two victoria-metrics (or vmstorage) processes appear in ps, or a Docker container and a systemd unit both point at the same volume.

Common Root Causes

• Two instances, one data dir — a manually launched process and the systemd unit, or two containers, mounting the same -storageDataPath.
• Overlapping restart — a new process starts before the old one has fully exited and released the lock.
• Stale lock after a crash — the host was hard-killed (OOM, power loss) and, in rare filesystem/NFS cases, the advisory lock reference lingers.
• Shared volume across replicas — a Kubernetes Deployment (not StatefulSet) scaling to 2 replicas mounts one ReadWriteOnce volume into two pods.
• NFS or network filesystem — flock semantics on NFS can misbehave, leaving the lock apparently held.
• Wrong data path — a second component (e.g. vmbackup/vmrestore misconfigured to write) targeting the live data directory.

Diagnostic Workflow

First find out whether a VictoriaMetrics process is genuinely still running against that directory. This is the single most important step — do not delete the lock file until you have proven no process holds it:

ps -ef | grep -E 'victoria-metrics|vmstorage' | grep -v grep
pgrep -a victoria-metrics

Identify exactly which process (if any) holds the lock file, using fuser/lsof on the actual flock.lock:

sudo fuser -v /victoria-metrics-data/flock.lock
sudo lsof /victoria-metrics-data/flock.lock

Confirm the configured data path so you are looking at the right directory:

curl -s http://<node>:8428/flags | grep -i storageDataPath   # if the old instance is still up
grep -i storageDataPath /etc/victoriametrics/*.conf 2>/dev/null
systemctl cat victoriametrics | grep -i storageDataPath

Read the service logs to see the exact FATAL line and timing:

journalctl -u victoriametrics --since '10 min ago' | grep -i 'lock\|FATAL'
journalctl -u vmstorage --since '10 min ago' | grep -i 'lock\|FATAL'

In containers, check for duplicate mounts of the same volume:

docker ps --format '{{.Names}}' | xargs -I{} sh -c 'echo {}; docker inspect {} --format "{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}"'

If fuser/lsof show no process holding the file, the lock is stale and safe to remove. If they show a live PID, that process is the owner — stop it, do not delete the file.

Example Root Cause Analysis

After a node reboot, vmstorage on one cluster shard refused to start, logging cannot obtain lock on file "/vmstorage-data/flock.lock": resource temporarily unavailable, and vmselect reported that shard unavailable on port 8401.

ps -ef | grep vmstorage revealed the answer: two vmstorage processes. The systemd unit had Restart=always, and during the messy reboot systemd had launched a new process while the old one — stuck flushing parts — was still alive and holding the flock. fuser -v /vmstorage-data/flock.lock showed the older PID owned the lock, so the new process could never acquire it and crash-looped.

The lock was doing its job: it prevented a second writer from corrupting the storage parts. The fix was to stop the unit cleanly, let systemd kill the stale process, confirm with fuser that nothing held the lock, and start once:

sudo systemctl stop vmstorage
sudo fuser -k /vmstorage-data/flock.lock   # only after confirming it is the stray old PID
sudo systemctl start vmstorage

vmstorage acquired the lock, opened 8482/8401/8400, and vmselect marked the shard healthy again. To prevent recurrence they set TimeoutStopSec high enough for a clean shutdown flush and ensured only the systemd unit — never a manual invocation — owned the directory.

Prevention Best Practices

• Give each data directory exactly one owner: manage VictoriaMetrics through systemd (or a single container) and never launch a second process by hand against the same -storageDataPath.
• In Kubernetes, run vmstorage and single-node VM as a StatefulSet with one PersistentVolume per pod, never a Deployment sharing a ReadWriteOnce volume.
• Set a generous TimeoutStopSec so a shutting-down instance can flush and release the lock before a restart is attempted.
• Avoid NFS/network filesystems for the data directory; local disk gives correct flock semantics and far better performance.
• Before any manual start, run pgrep victoria-metrics / fuser flock.lock to confirm nothing else owns the directory.
• Never delete flock.lock blindly — only remove it after lsof/fuser prove no process holds it.
• Alert on the service being down (target-down / scrape failure) so a crash-looping instance is caught early.

Quick Command Reference

# Is a VM process still running?
ps -ef | grep -E 'victoria-metrics|vmstorage' | grep -v grep
pgrep -a victoria-metrics

# Who holds the lock file?
sudo fuser -v /victoria-metrics-data/flock.lock
sudo lsof /victoria-metrics-data/flock.lock

# Confirm the configured data path
systemctl cat victoriametrics | grep -i storageDataPath

# Logs
journalctl -u victoriametrics --since '10 min ago' | grep -i 'lock\|FATAL'

# Clean restart (single owner). Only -k after confirming a stray old PID:
sudo systemctl stop victoriametrics
sudo fuser -k /victoria-metrics-data/flock.lock
sudo systemctl start victoriametrics

# Check for duplicate container mounts of the data volume
docker inspect <container> --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}'

Conclusion

cannot obtain lock on file "/victoria-metrics-data/flock.lock" is VictoriaMetrics protecting your data from two simultaneous writers. Resist the urge to delete the lock file first. Instead, prove whether a process still owns it with ps, fuser, and lsof: if a live instance holds it, stop that instance; only if nothing holds it is the lock stale and safe to clear. The lasting fix is architectural — one and only one process (a single systemd unit, a single StatefulSet pod) owns each data directory, backed by generous stop timeouts and local disk so shutdowns release the lock cleanly.