🚨 VictoriaMetrics Error: 'cannot create snapshot: invalid cross-device link' — Cause, Fix, and Troubleshooting Guide

Overview

VictoriaMetrics creates instant snapshots by making hard links from the live data directory into a snapshots/ subdirectory, which is fast and consumes almost no extra space. Hard links only work within a single filesystem, so if snapshots/ and the live data live on different mounts, the /snapshot/create call on single-node victoria-metrics (or vmstorage) fails with:

cannot create snapshot: cannot create hard links from "/victoria-metrics-data/data" to "/victoria-metrics-data/snapshots/20260712...": invalid cross-device link

The message is telling you exactly what happened: the source and destination sit on different devices, so the kernel refuses the hard link. VictoriaMetrics deliberately stores snapshots under -storageDataPath/snapshots; the failure almost always comes from someone mounting a separate volume over that subdirectory, or from auth/permission/disk problems that surface at the same step.

Symptoms

• GET /snapshot/create returns an error instead of a snapshot name.
• Logs show invalid cross-device link (errno EXDEV) referencing the snapshots/ path.
• Snapshots worked before a storage layout change and stopped after a new volume was mounted.
• /snapshot/create returns an authorization error rather than a link error when -snapshotAuthKey is involved.
• Backups driven by vmbackup fail because the underlying snapshot never gets created.

Common Root Causes

• Snapshots directory on a different filesystem — a separate volume is mounted over -storageDataPath/snapshots, so hard links from -storageDataPath/data cross a device boundary and fail.
• -snapshotAuthKey not set or mismatched — the request is rejected before or during creation because the key is missing or wrong.
• Permission denied on the data directory — the VictoriaMetrics user cannot create files under the snapshots path.
• Disk full — no space to write snapshot metadata, so creation aborts.

How to diagnose

Confirm whether the data and snapshots paths are actually on the same device. Different Mounted on (or different device numbers) is the smoking gun:

# Compare the filesystem/device for data vs snapshots
df -P /victoria-metrics-data/data /victoria-metrics-data/snapshots
stat -c '%d %n' /victoria-metrics-data/data /victoria-metrics-data/snapshots

Check ownership and free space on the storage path:

ls -ld /victoria-metrics-data /victoria-metrics-data/snapshots
df -h /victoria-metrics-data

Reproduce the snapshot call directly, passing the auth key if one is configured:

# Single-node snapshot API on 8428 (add ?authKey=... if -snapshotAuthKey is set)
curl -s 'http://localhost:8428/snapshot/create?authKey=REDACTED'

If df/stat show different devices for data and snapshots, the cross-device link is unavoidable until the layout is fixed — no flag will make a hard link span two filesystems.

Fixes

1. Keep snapshots on the same filesystem as -storageDataPath. VictoriaMetrics stores snapshots under -storageDataPath/snapshots by design. Do not mount a separate volume over that subdirectory. Mount the whole -storageDataPath on one volume so data/ and snapshots/ share a device:

./victoria-metrics -storageDataPath=/victoria-metrics-data
# /victoria-metrics-data/data and /victoria-metrics-data/snapshots must be one FS

2. Set and pass -snapshotAuthKey. Protect the snapshot API and use the matching key on every call:

./victoria-metrics -storageDataPath=/victoria-metrics-data -snapshotAuthKey=REDACTED
curl -s 'http://localhost:8428/snapshot/create?authKey=REDACTED'

3. Fix directory ownership and permissions. Ensure the process user owns the storage path so it can create the snapshot directory:

chown -R victoriametrics:victoriametrics /victoria-metrics-data
chmod 700 /victoria-metrics-data

4. Free disk space on the storage volume so snapshot metadata can be written (see the no-space guide).

5. Prune old snapshots to reclaim inodes and keep the directory manageable, either explicitly or via age-based cleanup:

curl -s 'http://localhost:8428/snapshot/list'
curl -s 'http://localhost:8428/snapshot/delete?snapshot=20260712...'
# Or expire automatically:
./victoria-metrics -storageDataPath=/victoria-metrics-data -snapshotsMaxAge=72h

What to watch out for

• Hard-link snapshots share data files with the live database — moving a snapshot to backup storage should be done with vmbackup, not by copying files while the process writes.
• A separate snapshots mount is the most common self-inflicted cause; audit your volume definitions in Docker/Kubernetes before blaming VictoriaMetrics.
• Snapshots hold references to data files, so old snapshots can pin disk space even after retention would otherwise free it — prune them with -snapshotsMaxAge or /snapshot/delete.
• Always pass the authKey when -snapshotAuthKey is set; a missing key produces an auth error that can be mistaken for the link failure.

Related

• VictoriaMetrics Error Guide: storageDataPath permission denied
• VictoriaMetrics Error Guide: no space left on device
• VictoriaMetrics Error Guide: read-only mode