🚨 Loki Error Guide: 'mkdir /loki: permission denied' — Fix Volume Ownership and fsGroup

Overview

Loki fails at startup when it cannot create the data directories it needs on a mounted volume. The Go runtime surfaces the underlying filesystem error directly:

error running loki: mkdir /loki: permission denied

The same failure appears for any subdirectory Loki tries to create on that mount:

error running loki: mkdir /loki/wal: permission denied

You will also see it on /loki/compactor, /loki/tsdb-cache, and other paths. The Loki container image runs as a non-root user (uid 10001 by default), but the mounted volume — a PVC, hostPath, or emptyDir remounted with wrong ownership — is owned by root. Since the process cannot write to a root-owned directory, the very first mkdir fails and Loki never starts. The fix is always about making the mount writable by the uid Loki actually runs as, not about running Loki as root.

Symptoms

• Loki crash-loops at startup with mkdir /loki: permission denied before any query or push is served.
• The failing path varies — /loki, /loki/wal, /loki/compactor, /loki/tsdb-cache — but the cause is identical.
• kubectl exec into the pod shows the mount owned by root:root while the process runs as uid 10001.
• The chart worked with an emptyDir but breaks after switching to a PVC or hostPath.
• Running the same image as root (for a quick test) starts cleanly, confirming a permissions problem.

Common Root Causes

• Volume owned by root, container runs non-root — the default image runs as uid 10001, but the PVC mounts as root:root and stays unwritable.
• Missing securityContext.fsGroup — without an fsGroup, Kubernetes does not group-own the volume to the pod, so the non-root user cannot write.
• hostPath with wrong ownership — a node-local directory created by root is mounted directly and never chowned to the Loki uid.
• Read-only mount — the volume is mounted readOnly: true (or the filesystem is read-only), so no directory can be created regardless of ownership.
• runAsUser overridden — a pod-level runAsUser that does not match the volume ownership or the fsGroup, leaving the process unable to write.

How to diagnose

1. Reproduce the parse/start offline to confirm the config itself is valid and the failure is purely filesystem, not configuration:

   loki -config.file=/etc/loki/config.yaml -verify-config

2. Read the startup logs to capture the exact path that failed:

   kubectl logs -l app=loki --tail=30 | grep 'permission denied'

3. Check where Loki writes in the config so you know which mounts must be writable:

   common:
     path_prefix: /loki
   ingester:
     wal:
       dir: /loki/wal
   compactor:
     working_directory: /loki/compactor

4. Inspect the mount ownership and mode from inside the pod against the uid Loki runs as:

   kubectl exec deploy/loki -- sh -c 'id; ls -ldn /loki /loki/wal 2>&1'

5. Confirm the volume is not read-only by checking the volumeMount and the effective mount flags:

   kubectl exec deploy/loki -- sh -c 'touch /loki/.writetest && echo writable || echo READ-ONLY'

Fixes

Set fsGroup (and runAsUser) so the PVC is group-writable by the Loki uid. This is the cleanest fix: Kubernetes recursively sets the volume group to fsGroup, and the non-root process can create its directories:

spec:
  securityContext:
    runAsUser: 10001
    runAsGroup: 10001
    fsGroup: 10001

Use an initContainer to chown the path when fsGroup is not honoured (some CSI drivers) or for a hostPath. Run it as root once to hand ownership to the Loki uid before the main container starts:

initContainers:
  - name: chown-loki
    image: busybox:1.36
    command: ["sh", "-c", "chown -R 10001:10001 /loki"]
    securityContext:
      runAsUser: 0
    volumeMounts:
      - name: storage
        mountPath: /loki

Ensure the volume is actually writable — drop any readOnly: true on the data mount, since Loki must create and rotate files there:

volumeMounts:
  - name: storage
    mountPath: /loki
    readOnly: false

Point the writable paths at the mounted volume so the WAL, compactor, and caches all live under a directory Loki owns, rather than an unwritable default:

common:
  path_prefix: /loki
ingester:
  wal:
    enabled: true
    dir: /loki/wal
compactor:
  working_directory: /loki/compactor

Fix a hostPath’s node-level ownership directly when you must use one, so the directory the kubelet mounts is already owned by the Loki uid:

# on the node hosting the pod
sudo mkdir -p /data/loki && sudo chown -R 10001:10001 /data/loki

What to watch out for

• Do not “fix” this by running Loki as root — it defeats the image’s non-root hardening and hides the real ownership problem.
• fsGroup chowns the whole volume recursively on mount, which can be slow for large existing volumes; expect a delayed first start.
• Some CSI drivers ignore fsGroup (or need fsGroupChangePolicy); if it does not take effect, fall back to an initContainer chown.
• Every writable path Loki uses must resolve to the owned mount — a stray default like a WAL dir outside the volume will still fail even after you fix /loki.
• A readOnly: true mount produces the identical error regardless of ownership, so check the mount flags before chasing uid mismatches.

Related

• Loki Error Guide: ‘failed to flush chunks’ — the write-path failure that follows once Loki starts but still cannot persist data.
• Loki Error Guide: ‘access denied’ to the object store — the object-store equivalent of a local permission problem.
• Loki Error Guide: ‘empty ring’ — a ring failure that often accompanies ingesters unable to start on their volume.