🚨 VictoriaMetrics Error: 'mkdir /victoria-metrics-data: permission denied' — Cause, Fix, and Troubleshooting Guide

Overview

On startup, single-node victoria-metrics (or vmstorage) must create and write its data directory at -storageDataPath. If the process user cannot create that directory on the mounted volume, VictoriaMetrics fails fast before it ever serves traffic:

cannot create directory "/victoria-metrics-data": mkdir /victoria-metrics-data: permission denied

This is an operating-system permission problem, not a VictoriaMetrics bug. The container or process is running as a user that lacks write access to the target path — usually because a bind mount or persistent volume is owned by a different UID, or because SELinux/AppArmor or a read-only mount blocks the write.

Symptoms

• The process exits immediately at startup with permission denied on the -storageDataPath.
• A container restart-loops (CrashLoopBackOff in Kubernetes) and never becomes ready.
• The error appears right after switching to a new bind mount, PVC, or a rebuilt host directory.
• victoria-metrics runs fine as root but fails when dropped to a non-root user.
• Writes are refused even though the path exists and looks correct in ls.

Common Root Causes

• Container/process user cannot write the mounted volume — the image runs as a non-root UID that has no write bit on the mount.
• Wrong ownership on a bind mount — the host directory is owned by root (or another UID) while the process runs as a different user.
• SELinux or AppArmor denial — the volume is not labeled for container access, so the kernel blocks the write.
• Read-only mount — the volume is mounted ro, so no directory can be created regardless of ownership.

How to diagnose

Confirm which UID/GID the process runs as and who owns the path:

# UID the container process runs as
docker inspect --format '{{.Config.User}}' victoriametrics
# Ownership and mode of the host path
ls -ld /srv/victoria-metrics-data

Check whether the mount is read-only and look for SELinux denials:

mount | grep victoria-metrics-data      # look for "ro" in the options
# On SELinux hosts, denials show up in the audit log:
ausearch -m avc -ts recent | grep victoria-metrics-data

In Kubernetes, describe the pod and inspect the securityContext and volume:

kubectl describe pod victoriametrics-0
kubectl get pod victoriametrics-0 -o jsonpath='{.spec.securityContext}'

Fixes

1. Chown the volume to the UID VictoriaMetrics runs as. Match the host directory owner to the container user:

# Example: image runs as uid 1000
chown -R 1000:1000 /srv/victoria-metrics-data

2. Set correct mode on the host bind mount so the process user can write:

chmod 755 /srv/victoria-metrics-data

3. Add the SELinux relabel option to the volume so the container may write it:

docker run -v /srv/victoria-metrics-data:/victoria-metrics-data:Z \
  victoriametrics/victoria-metrics
# use :z instead of :Z when the volume is shared between containers

4. Ensure the mount is read-write. Remove any :ro / readOnly: true on the data volume — VictoriaMetrics must write its storage path.

5. Run with a matching user/securityContext in Kubernetes. Use fsGroup so the mounted PVC is group-owned by the process, and set runAsUser to match:

securityContext:
  runAsUser: 1000
  runAsGroup: 1000
  fsGroup: 1000

What to watch out for

• Changing the image to run as root “fixes” it but leaves data owned by root; if you later drop privileges the same error returns. Fix ownership instead.
• fsGroup only relabels the volume for the pod’s group — files created by other UIDs still need to be group-readable/writable.
• On SELinux hosts, forgetting :Z/:z is the single most common cause; the path can look perfectly owned yet still be denied.
• A read-only root filesystem is fine as long as the -storageDataPath volume itself is mounted read-write.

Related

• VictoriaMetrics Error Guide: read-only mode
• VictoriaMetrics Error Guide: no space left on device
• VictoriaMetrics Error Guide: cannot obtain lock