🚨 VictoriaMetrics Error Guide: 'insufficient free disk space; switching to read-only mode' — Free or Grow the Storage Disk

Overview

VictoriaMetrics guards against filling its disk completely by refusing new writes once free space drops below a configured floor. When that floor is crossed, vmstorage (or single-node VM) logs the switch and rejects ingestion:

insufficient free disk space at "/vmstorage-data" (1073741824 bytes left, -storage.minFreeDiskSpaceBytes=10737418240 bytes required); switching to read-only mode

On single-node VictoriaMetrics the same message references the single data path:

insufficient free disk space at "/victoria-metrics-data"; switching to read-only mode; deleting data or increasing disk size is required to switch back

This is a deliberate protection, not corruption. Running a metrics database onto a 100%-full disk risks unrecoverable damage during compaction and merges, so VictoriaMetrics stops accepting new samples while it still has a safety margin (-storage.minFreeDiskSpaceBytes, default 10GB). Reads and queries keep working; only writes are blocked. To recover you must give it room — free space, grow the disk, or reduce retention — after which it automatically returns to read-write.

Symptoms

• Ingestion stops: vminsert/scrapers report write rejections, and metrics stop advancing while old data still queries fine.
• Logs show switching to read-only mode on vmstorage (port 8482) or single-node VM (port 8428).
• vm_storage_is_read_only gauge reads 1; vm_free_disk_space_bytes sits at or below -storage.minFreeDiskSpaceBytes.
• Grafana panels show a flat line / gap starting at the moment the mode switched.
• df -h on the storage volume shows it near or at 100% used.
• The condition recovers on its own once free space rises back above the threshold — no restart needed.

Common Root Causes

• Disk genuinely full — retention is too long for the ingested volume, and the data directory grew past the disk.
• Cardinality explosion — a bad label (request IDs, timestamps, full URLs) multiplied active series, inflating index and data size fast.
• Retention misconfigured — -retentionPeriod set higher than the disk can hold, or never set (long default retention).
• Undersized volume — the disk was provisioned before ingestion volume grew.
• Compaction/merge headroom consumed — big merges temporarily need extra space; a too-small -storage.minFreeDiskSpaceBytes cut it too fine.
• Other files on the same volume — logs, backups, or unrelated data sharing the storage disk and eating the margin.
• Downsampling not enabled — high-resolution raw data kept forever when coarser long-term data would suffice.

Diagnostic Workflow

Confirm the mode and the free-space numbers straight from the metrics endpoint. On vmstorage (port 8482) or single-node (8428):

curl -s http://<node>:8428/metrics | grep -E 'vm_storage_is_read_only|vm_free_disk_space_bytes|vm_data_size_bytes'

vm_storage_is_read_only 1 confirms the switch; compare vm_free_disk_space_bytes against the configured floor:

curl -s http://<node>:8428/flags | grep -i 'minFreeDiskSpaceBytes\|retentionPeriod\|storageDataPath'

Check the actual filesystem — bytes and, on huge cardinality, inodes:

df -h /victoria-metrics-data
df -i /victoria-metrics-data
sudo du -sh /victoria-metrics-data/* | sort -rh | head

Find what is consuming space. If cardinality is the driver, the index directory grows disproportionately; confirm via TSDB status:

curl -s 'http://<node>:8428/api/v1/status/tsdb' | head -40
curl -s http://<node>:8428/metrics | grep -E 'vm_rows |vm_cache_size_bytes'

Read the logs for the exact threshold and timing:

journalctl -u victoriametrics --since '30 min ago' | grep -i 'read-only\|free disk'
journalctl -u vmstorage --since '30 min ago' | grep -i 'read-only\|free disk'

Verify whether other, non-VM files share the volume and could be freed:

sudo du -sh /path/to/volume/* | sort -rh | head

Example Root Cause Analysis

An on-call engineer noticed metrics flatlined across every dashboard at 02:14, though historical queries still worked. curl http://vmstorage-0:8482/metrics | grep vm_storage_is_read_only returned 1, and vm_free_disk_space_bytes sat at 1073741824 (1GB) against -storage.minFreeDiskSpaceBytes=10737418240 (10GB) — VictoriaMetrics had switched to read-only exactly as designed.

df -h /vmstorage-data showed 99% used. du -sh on the data path showed the index directory had ballooned. /api/v1/status/tsdb confirmed it: a recently deployed service was labeling metrics with a per-request trace_id, exploding active series from ~400k to ~9M in a day. vm_rows growth had gone vertical.

The disk itself was correctly sized for the intended retention; the real defect was cardinality. Immediate recovery came from growing the cloud volume and running growpart + resize2fs, which pushed free space above 10GB and VictoriaMetrics automatically returned to read-write with no restart. The permanent fix was dropping the trace_id label at the scrape/relabel stage and setting a -search.maxUniqueTimeseries guard. Retention was also reviewed so the disk had months of headroom for the corrected series count.

Prevention Best Practices

• Alert on vm_free_disk_space_bytes approaching -storage.minFreeDiskSpaceBytes (e.g. warn at 2x the floor) and on vm_storage_is_read_only == 1, so you act before writes stop.
• Set -retentionPeriod deliberately to match disk capacity and ingested volume; don’t rely on defaults.
• Keep -storage.minFreeDiskSpaceBytes at a healthy margin (default 10GB, more for large deployments) so compaction/merges always have working room.
• Control cardinality with relabeling: drop high-entropy labels (trace/request IDs, timestamps, raw URLs) before they hit storage, and cap with -search.maxUniqueTimeseries.
• Enable downsampling / configure retention filters so long-term data is coarser and smaller than raw high-resolution samples (VictoriaMetrics enterprise downsampling or reduced retention).
• Give vmstorage its own dedicated volume — no logs, backups, or unrelated files competing for the margin.
• Provision cloud disks that can be grown online (growpart/resize2fs/xfs_growfs) so recovery is a resize, not a rebuild.

Quick Command Reference

# Confirm read-only state and free-space numbers
curl -s http://<node>:8428/metrics | grep -E 'vm_storage_is_read_only|vm_free_disk_space_bytes'

# Show the free-space floor and retention flags
curl -s http://<node>:8428/flags | grep -i 'minFreeDiskSpaceBytes\|retentionPeriod'

# Filesystem usage (bytes and inodes)
df -h /victoria-metrics-data
df -i /victoria-metrics-data
sudo du -sh /victoria-metrics-data/* | sort -rh | head

# Cardinality check (usual root cause of runaway growth)
curl -s 'http://<node>:8428/api/v1/status/tsdb' | head -40

# Logs
journalctl -u vmstorage --since '30 min ago' | grep -i 'read-only\|free disk'

# Recover by growing the disk online (cloud resize first), then:
sudo growpart /dev/nvme0n1 1
sudo resize2fs /dev/nvme0n1p1        # or xfs_growfs /vmstorage-data

# Or reduce retention to reclaim space (flag)
-retentionPeriod=3

Conclusion

insufficient free disk space ...; switching to read-only mode is VictoriaMetrics protecting your data by stopping writes before the disk hits 100%. Reads keep working, and the moment vm_free_disk_space_bytes climbs back above -storage.minFreeDiskSpaceBytes it returns to read-write on its own — no restart. Recover by freeing space, growing the volume, or trimming retention, then find the real driver: nine times out of ten it is cardinality growth, not honest data volume. Alert on free-space and the vm_storage_is_read_only gauge, control labels with relabeling, and size disks with compaction headroom so this guardrail never trips in the first place.