🚨 VictoriaMetrics Error Guide: 'no space left on device' — Reclaim and Cap Disk

Overview

When the filesystem behind a VictoriaMetrics data directory or a vmagent persistent queue fills up, the OS returns ENOSPC and the process surfaces it in the logs while writes fail:

cannot write data to file "/victoria-metrics-data/data/small/2026_07/...": no space left on device

On vmagent the same underlying condition looks like:

cannot save persistent queue "/vmagent-remotewrite-data/...": write ...: no space left on device

This is different from VictoriaMetrics’ graceful read-only mode (which it enters proactively at -storage.minFreeDiskSpaceBytes to protect the database). no space left on device is the hard, OS-level failure you hit when the disk actually reaches zero — at which point samples are dropped and, on vmstorage, background merges can stall.

Symptoms

• vmstorage or single-node victoria-metrics logs repeat no space left on device and ingestion stops persisting.
• vmagent logs show it cannot write to its -remoteWrite.tmpDataPath persistent queue, and buffered data is lost.
• df -h on the data volume shows 100% used (or inodes exhausted at 100% IUse%).
• Gaps appear in recent data; on cluster mode, affected vmstorage nodes may drop out of vminsert’s healthy set.
• Background merges (vm_pending_rows, part counts) climb because merging needs free scratch space it doesn’t have.

Common Root Causes

• Retention too long for the volume — -retentionPeriod outgrew the disk as ingestion or cardinality rose.
• A cardinality/churn explosion — a label leak multiplied series count, inflating on-disk data far faster than expected.
• Unbounded vmagent queue — -remoteWrite.maxDiskUsagePerURL unset, so a downstream outage let the persistent queue grow until it filled the disk.
• No headroom for merges — vmstorage needs free space to merge parts; a volume run near-full stalls compaction and then fails writes.
• Shared volume — logs, another service, or a snapshot on the same filesystem consumed the space.
• Inode exhaustion — many small files (parts, queue segments) exhausted inodes even though bytes remained.

How to diagnose

Confirm it is genuinely a full filesystem and find what is consuming it:

# Bytes and inodes on the data volume
df -h /victoria-metrics-data
df -i /victoria-metrics-data

# Largest consumers under the data dir
du -h --max-depth=1 /victoria-metrics-data | sort -rh | head

# vmagent persistent-queue size
du -sh /vmagent-remotewrite-data

Check VictoriaMetrics’ own view of free space and read-only state:

curl -s http://localhost:8428/metrics | grep -E 'vm_free_disk_space_bytes|vm_storage_is_read_only'

Look at snapshots — an old snapshot pins disk that a retention change won’t reclaim:

curl -s http://localhost:8428/snapshot/list

Fixes

1. Get free space back immediately. Delete stale snapshots (they hold hardlinks to old parts):

# Remove a specific snapshot, or all of them
curl -s 'http://localhost:8428/snapshot/delete?snapshot=<name>'
curl -s http://localhost:8428/snapshot/delete_all

If a vmagent queue filled the disk, cap it so it can no longer grow without bound (excess is dropped instead of filling the disk):

./vmagent -remoteWrite.url=http://vminsert:8480/insert/0/prometheus/api/v1/write \
  -remoteWrite.tmpDataPath=/vmagent-remotewrite-data \
  -remoteWrite.maxDiskUsagePerURL=10GB

2. Reduce what is stored. Shorten retention so old data is dropped on the next background cycle:

./victoria-metrics-prod -storageDataPath=/victoria-metrics-data -retentionPeriod=3

3. Fix the cardinality at the source. If series count exploded, drop the offending label at ingest with metric_relabel_configs (see the high-churn guide) so on-disk growth slows.

4. Grow or move the volume. Expand the block device / filesystem, or move -storageDataPath to a larger disk. Set -storage.minFreeDiskSpaceBytes so VictoriaMetrics switches to read-only before the disk hits zero next time:

./victoria-metrics-prod -storageDataPath=/victoria-metrics-data \
  -storage.minFreeDiskSpaceBytes=20GB

What to watch out for

• Deleting snapshots frees space asynchronously; give background cleanup a moment, and re-check df -h before assuming it didn’t work.
• Shortening -retentionPeriod reclaims space on the next retention cycle, not instantly — snapshot deletion and a larger volume are the fast levers.
• Set -remoteWrite.maxDiskUsagePerURL on every vmagent; without it, a long downstream outage can fill the disk and take the agent down.
• Prefer letting VictoriaMetrics hit read-only mode (via -storage.minFreeDiskSpaceBytes) over ever reaching a true no space left on device — read-only is recoverable and protects the database; a full disk during a merge is not graceful.
• Watch both bytes and inodes; a volume can hit 100% inodes while showing free bytes.

Related

• VictoriaMetrics Error Guide: read-only mode
• VictoriaMetrics Error Guide: OOM killed
• VictoriaMetrics Error Guide: too many open files