🚨 Kubernetes Error Guide: 'MountVolume.SetUp failed for volume' Storage Mount Failures

Exact Error Message

MountVolume.SetUp failed is a kubelet event raised when it cannot make a volume available inside a pod’s filesystem. It appears in kubectl describe pod events, in several variants depending on the underlying cause:

Warning  FailedMount  18s (x7 over 2m13s)  kubelet  MountVolume.SetUp failed for volume "app-config" : configmap "app-config" not found

Warning  FailedMount  31s  kubelet  MountVolume.SetUp failed for volume "tls-secret" : secret "tls-secret" not found

Warning  FailedMount  44s  kubelet  MountVolume.SetUp failed for volume "data" : rpc error: code = Internal desc = volume attachment is being deleted; csi.NodeStageVolume failed

Warning  FailedMount  12s  kubelet  MountVolume.SetUp failed for volume "app-config" : failed to sync configmap cache: timed out waiting for the condition

Warning  FailedMount  9s  kubelet  MountVolume.SetUp failed for volume "data" : mount failed: ... permission denied

The pod stays in ContainerCreating and never reaches Running because the container cannot start until all of its volumes are mounted.

What the Error Means

When a pod is scheduled, the kubelet must set up every volume in .spec.volumes before it starts any container. For each volume it runs a SetUp operation: fetching the Secret/ConfigMap object and writing it to a tmpfs, calling the CSI driver to stage and publish a block/file volume, or projecting a ServiceAccount token. If any of these steps fails, the kubelet emits MountVolume.SetUp failed for volume "<name>" with the underlying reason appended, retries with backoff, and keeps the pod in ContainerCreating.

The text after the final colon is the real diagnosis. configmap/secret "..." not found is an object-reference problem. rpc error: code = ... comes from the CSI driver. permission denied is a filesystem ownership/fsGroup issue. failed to sync ... cache: timed out is a kubelet informer/cache problem, often transient or RBAC-related. Read that suffix first.

This is closely related to, but distinct from, a PVC that never binds. If your PersistentVolumeClaim is stuck Pending, the pod fails to schedule at all — see the PVC not bound guide. MountVolume.SetUp failed happens after scheduling, when the volume exists (or should) but cannot be mounted.

Common Causes

• Referenced Secret or ConfigMap missing. The pod references a Secret/ConfigMap that was never created, was deleted, or lives in a different namespace (volume sources are namespace-local).
• CSI driver or node plugin not running. The CSI node DaemonSet pod on that node is crashing or absent, so NodeStageVolume/NodePublishVolume RPCs fail.
• Stuck attach/detach. A VolumeAttachment is wedged (e.g. node lost, volume still attached to an old node), so the new mount cannot proceed.
• Wrong fsGroup or filesystem permissions. The mounted filesystem’s ownership does not match the container’s UID/fsGroup, yielding permission denied.
• Backend mount failure. NFS server unreachable, EBS volume in the wrong AZ, or the underlying device fails to attach.
• subPath / readOnly issues. A subPath referencing a non-existent path, or a read-only mount where the app expects to write.
• Projected ServiceAccount token problems. Audience/expiry misconfiguration on a projected token source.
• Cache sync timeout. The kubelet cannot populate its Secret/ConfigMap informer cache, often because the node’s credentials lack get/watch or the API server is slow.

How to Reproduce the Error

Create a pod that mounts a ConfigMap that does not exist:

apiVersion: v1
kind: Pod
metadata:
  name: config-demo
  namespace: prod
spec:
  containers:
  - name: app
    image: busybox:1.36
    command: ["sleep", "3600"]
    volumeMounts:
    - name: app-config
      mountPath: /etc/app
  volumes:
  - name: app-config
    configMap:
      name: app-config        # never created

kubectl apply -f config-demo.yaml
kubectl describe pod config-demo -n prod | grep -A4 Events

Warning  FailedMount  10s  kubelet  MountVolume.SetUp failed for volume "app-config" : configmap "app-config" not found

The pod is stuck in ContainerCreating until the ConfigMap is created in the prod namespace.

Diagnostic Commands

Start with the pod events — they name the volume and the reason:

kubectl describe pod <POD> -n <NS> | grep -A8 Events
kubectl get pod <POD> -n <NS> -o jsonpath='{.spec.volumes}' | jq

Confirm the referenced Secret/ConfigMap exists in the same namespace:

kubectl get configmap,secret -n <NS> | grep -E 'app-config|tls-secret'

For CSI / block storage, inspect the attachment and the PV:

kubectl get volumeattachment | grep <PV_NAME>
kubectl get pvc -n <NS>
kubectl get pv <PV_NAME> -o jsonpath='{.spec.nodeAffinity}'

Check the CSI node plugin pod on the affected node:

NODE=$(kubectl get pod <POD> -n <NS> -o jsonpath='{.spec.nodeName}')
kubectl get pods -n kube-system -o wide | grep -E "csi.*$NODE"
kubectl logs -n kube-system <csi-node-pod> -c csi-driver --tail=50

Go to the kubelet itself for low-level mount errors:

ssh <NODE> 'sudo journalctl -u kubelet --since "5 min ago" | grep -i mount'

kubelet: E0625 mount_linux.go: mount failed: exit status 32
kubelet: Output: mount.nfs: Connection timed out

Step-by-Step Resolution

Missing Secret / ConfigMap

Create the object in the pod’s namespace, then recreate the pod (the kubelet retries automatically once it appears):

kubectl create configmap app-config --from-file=app.conf -n prod
# or for a secret
kubectl create secret generic tls-secret --from-file=tls.crt --from-file=tls.key -n prod
kubectl delete pod config-demo -n prod   # or let a Deployment reconcile

If it exists but in the wrong namespace, move/recreate it — volume sources cannot cross namespaces.

CSI node plugin down

Restart the CSI node DaemonSet and verify it lands on the node:

kubectl rollout restart daemonset/<csi-node-driver> -n kube-system
kubectl get pods -n kube-system -o wide | grep csi | grep <NODE>

Stuck VolumeAttachment

When a VolumeAttachment is wedged after a node loss, confirm the volume is truly detached at the cloud level, then remove the stale attachment:

kubectl get volumeattachment | grep <PV_NAME>
kubectl delete volumeattachment <ATTACHMENT_NAME>

Only delete after verifying (in AWS/GCP console or CLI) the volume is not still attached to a live node, to avoid multi-attach corruption.

Permission denied / fsGroup

Set fsGroup so the kubelet chowns the volume to a group the container can access:

spec:
  securityContext:
    fsGroup: 2000
    fsGroupChangePolicy: OnRootMismatch
  containers:
  - name: app
    securityContext:
      runAsUser: 1000
      runAsGroup: 2000

NFS / EBS backend failure

Test reachability and AZ placement. For EBS, the PV’s node affinity must match the pod’s node AZ:

# from the node: confirm NFS server is reachable
ssh <NODE> 'showmount -e <NFS_SERVER>'
# confirm EBS PV AZ matches node AZ
kubectl get pv <PV_NAME> -o jsonpath='{.spec.nodeAffinity}'
kubectl get node <NODE> -o jsonpath='{.metadata.labels.topology\.kubernetes\.io/zone}'

subPath does not exist

Ensure the subPath already exists in the source volume, or remove subPath and mount the whole volume — the kubelet does not create missing subPaths for ConfigMap/Secret sources.

Prevention and Best Practices

• Deploy referenced Secrets and ConfigMaps before (or atomically with) the workloads that mount them — use Helm hooks or Kustomize ordering so the object exists when the pod schedules.
• Keep CSI node DaemonSets healthy and alert on their readiness; a single crashed CSI pod blocks every new volume mount on that node.
• Set fsGroup (with fsGroupChangePolicy: OnRootMismatch to avoid slow recursive chowns) for any non-root workload that writes to a mounted volume.
• Pin StatefulSet pods and their PVs to the same topology zone so EBS-style volumes are always attachable where the pod lands.
• Monitor for stale VolumeAttachment objects after node failures and reconcile them carefully against the cloud provider’s actual attachment state. See more in Kubernetes & Helm guides.

Related Errors

• pod has unbound immediate PersistentVolumeClaims — the PVC never bound, so the pod cannot even schedule. This precedes mounting. See the PVC not bound guide.
• Multi-Attach error for volume "..." — the volume is still attached to another node; a stuck VolumeAttachment blocks the new mount.
• Unable to attach or mount volumes: timed out waiting for the condition — the umbrella timeout the pod shows after repeated MountVolume.SetUp failed events.

Frequently Asked Questions

Why does my pod stay in ContainerCreating instead of crashing? The kubelet must mount every volume before starting any container. If MountVolume.SetUp keeps failing, there is no container to crash — the pod sits in ContainerCreating and the kubelet retries the mount with backoff until the volume becomes available or the pod is deleted.

The Secret exists but I still get “secret not found” — why? Volume sources are namespace-local. The kubelet looks for the Secret in the pod’s namespace. If your Secret is in default but the pod runs in prod, the lookup fails. Recreate the Secret in the pod’s namespace; you cannot reference a Secret across namespaces in a volume mount.

How do I tell a CSI problem from a Secret/ConfigMap problem? Read the text after the volume name. configmap/secret "..." not found is an object-reference issue you fix with kubectl create. Anything starting with rpc error: code = ... comes from the CSI driver — check the CSI node plugin pod logs and the VolumeAttachment. permission denied is a filesystem/fsGroup issue.

Is this the same as a PVC stuck in Pending? No. A Pending PVC means no PersistentVolume could be bound, so the pod fails to schedule. MountVolume.SetUp failed happens after scheduling, when the volume should be mountable but the kubelet cannot complete the mount. See the PVC not bound guide for the binding case.

What causes “failed to sync secret cache: timed out”? The kubelet populates an informer cache for Secrets/ConfigMaps. A timeout usually means the API server was slow or the node’s credentials lack get/watch on that object. It is often transient and clears on retry; if persistent, check API server health and the kubelet’s RBAC.