☸️ Kubernetes Error Guide: 'permission denied' on a Volume Despite fsGroup

Overview

You set securityContext.fsGroup expecting Kubernetes to grant your non-root container write access to a mounted volume, yet the application still logs permission denied when it tries to write. This is one of the most confusing storage errors in Kubernetes because the fix you applied looks correct but did not take effect.

fsGroup tells the kubelet to recursively change group ownership and set the setgid bit on the volume so processes in that supplemental group can write. But several things can stop that from happening: the CSI driver may not support fsGroup ownership changes, fsGroupChangePolicy may skip a volume that already has “wrong” ownership, or the backend (NFS, an SMB share, or a read-only volume) may simply not honor POSIX group ownership at all.

The result is a running pod that cannot write its data directory. This is a permissions-and-backend problem, not an application bug.

Symptoms

Error: EACCES: permission denied, open '/data/app.db'
...
Warning  Failed  kubelet  Error: failed to start container "app":
container process caused: chdir to cwd "/data": permission denied

The pod may reach Running but crash-loop, or the app logs write failures. kubectl exec into the container and touch /data/test returns Permission denied, and ls -ld /data shows ownership that does not match your fsGroup.

Common Root Causes

1. CSI driver does not apply fsGroup

Some CSI drivers advertise fsGroupPolicy: None, meaning Kubernetes will not change volume ownership. The fsGroup you set is silently ignored for that volume.

2. fsGroupChangePolicy skipped the volume

With fsGroupChangePolicy: OnRootMismatch, the kubelet only fixes ownership when the volume root’s group differs from the expected value. If the root already looks “right” but subdirectories do not, they stay unwritable.

3. NFS / SMB backend ignores POSIX group ownership

Network filesystems frequently enforce ownership on the server side (root squash, fixed uid/gid mapping). Changing group ownership from the client has no effect, so fsGroup cannot grant access.

4. Read-only volume or read-only root filesystem

A volume mounted read-only, or securityContext.readOnlyRootFilesystem: true combined with an unwritable mount, produces permission denied regardless of group.

Diagnostic Workflow

Step 1: Confirm the fsGroup the pod actually requested

kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.securityContext}'

Verify fsGroup is set at the pod level (not just container level, where it is ignored).

Step 2: Inspect ownership on the mount from inside the pod

kubectl exec -it <pod> -n <namespace> -- ls -ld /data
kubectl exec -it <pod> -n <namespace> -- id

Compare the directory’s group to the fsGroup and to the process’s supplemental groups.

Step 3: Check the CSI driver’s fsGroupPolicy

kubectl get csidriver <driver-name> -o jsonpath='{.spec.fsGroupPolicy}'

None means fsGroup ownership changes are not applied by Kubernetes.

Step 4: Determine the backend type

kubectl get pv <pv-name> -o jsonpath='{.spec.csi.driver}{" "}{.spec.nfs}'

An NFS source, or a driver known to defer to the server, points to a backend-ownership root cause.

Step-by-Step Resolution

1. Ensure fsGroup is at the pod level and add fsGroupChangePolicy: Always so ownership is fixed even when the root looks correct:

   spec:
     securityContext:
       fsGroup: 2000
       fsGroupChangePolicy: Always

2. If the CSI driver reports fsGroupPolicy: None, Kubernetes will not change ownership. Set ownership yourself with an init container:

   initContainers:
     - name: fix-perms
       image: busybox
       command: ["sh", "-c", "chown -R 2000:2000 /data"]
       securityContext:
         runAsUser: 0
       volumeMounts:
         - name: data
           mountPath: /data

3. For NFS or SMB backends, fix permissions on the server (export as the right uid/gid, disable root squash where appropriate) or set mountOptions on the PV to map ownership. Client-side fsGroup cannot override server enforcement.

4. Confirm the volume is not mounted read-only:

   kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.volumes}' | grep -i readOnly

   Remove readOnly: true from the volume or volumeMount if the app must write.

5. Recreate the pod so the kubelet re-applies ownership:

   kubectl delete pod <pod> -n <namespace>

6. Verify the app can now write:

   kubectl exec -it <pod> -n <namespace> -- touch /data/test && echo ok

   ok

Prevention

• Always set fsGroup at the pod-level securityContext, not inside a container, where it has no effect.
• Prefer fsGroupChangePolicy: Always for volumes whose subdirectory ownership you cannot guarantee, accepting the extra chown cost on mount.
• Check a CSI driver’s fsGroupPolicy before relying on fsGroup; if it is None, plan for an init container or image-level ownership.
• For NFS/SMB, standardize uid/gid mapping between the cluster and the storage server so ownership is consistent from the start.
• Review securityContext and volume settings in CI. The DevOps AI prompt library includes prompts for auditing pod security contexts and volume permissions.

Related Errors

• MountVolume.SetUp failed ... permission denied — a mount-time permission failure rather than an app-write failure.
• chown: Operation not permitted — the kubelet or init container cannot change ownership, common on root-squashed NFS.
• read-only file system — the volume or root filesystem is mounted read-only, a distinct cause of write failures.
• container has runAsNonRoot and image will run as root — a related securityContext validation error.

Frequently Asked Questions

Why does fsGroup not fix my permissions? Either the CSI driver has fsGroupPolicy: None and ignores it, fsGroupChangePolicy: OnRootMismatch skipped the volume, or the backend (NFS/SMB) enforces ownership server-side. Each requires a different fix.

Where should fsGroup be set? At the pod-level spec.securityContext.fsGroup. Setting it inside a container’s securityContext does nothing for volume ownership.

What does fsGroupChangePolicy: Always do? It forces the kubelet to recursively set group ownership and the setgid bit on every mount, even when the volume root already appears correct. Use it when subdirectory ownership is inconsistent.

How do I fix NFS permission denied? Fix it on the NFS server: export with the correct uid/gid, avoid root squash for the workload, or map ownership via mount options. Client-side fsGroup cannot override server-side enforcement.

Is an init container a safe workaround? Yes, running an init container as root to chown the mount is a common, reliable pattern when the CSI driver will not apply fsGroup. For more storage guidance, see the Kubernetes & Helm guides.