🚨 Kubernetes Error Guide: 'Warning FailedMount' MountVolume.SetUp Failed Event

Exact Error Message

A scheduled pod sticks in ContainerCreating and the event stream carries a Warning FailedMount event naming a specific volume that the kubelet could not set up:

Events:
  Type     Reason       Age                 From     Message
  ----     ------       ----                ----     -------
  Warning  FailedMount  18s (x6 over 41s)   kubelet  MountVolume.SetUp failed for volume "app-config" : configmap "app-config" not found

The same FailedMount reason fronts several distinct underlying problems:

MountVolume.SetUp failed for volume "tls-certs" : secret "tls-certs" not found
MountVolume.SetUp failed for volume "data" : rpc error: code = Internal desc = mount failed: subPath "conf.d" not found
MountVolume.SetUp failed for volume "data" : mount failed: permission denied

The pod stays ContainerCreating while the event repeats with a climbing (xN over ...) counter:

NAME     READY   STATUS              RESTARTS   AGE
web-0    0/1     ContainerCreating   0          47s

FailedMount with MountVolume.SetUp failed is the per-volume event: the kubelet names exactly which volume failed and why, and retries on a backoff.

What the Error Means

FailedMount is the kubelet event emitted whenever a single volume fails its setup step. For projected volumes — secret, configMap, downwardAPI, projected — “setup” means materializing the data into a tmpfs directory and bind-mounting it into the container. For CSI/persistent volumes it means the NodePublishVolume (and subPath) step. When any of these fail, the kubelet records MountVolume.SetUp failed for volume "<name>" followed by the specific reason, and keeps retrying with exponential backoff.

The crucial detail is that the message is per-volume and specific — it names the volume and the exact failure (configmap "x" not found, subPath ... not found, permission denied). This is different from the kubelet’s aggregate timeout message (Unable to attach or mount volumes ... timed out waiting for the condition), which only appears later and lists volume names without the root cause. The FailedMount event is where the actual reason lives — read it first.

For attach-stage and timeout-aggregate failures, see the companion guide on Unable to attach or mount volumes. This page focuses on the FailedMount event itself and the config-volume causes that dominate it.

Common Causes

• Missing configMap — configmap "x" not found: the volume references a configMap that does not exist in the pod’s namespace.
• Missing secret — secret "x" not found: same, for a secret volume or projected secret source.
• Wrong key in optional=false source — a referenced items key does not exist in the secret/configMap.
• subPath not found — subPath "conf.d" not found: the subPath points at a file/dir that is not present in the volume.
• Permission denied — fsGroup/runAsUser cannot read the mounted files, or SELinux relabeling blocks access.
• Namespace mismatch — the secret/configMap exists, but in a different namespace than the pod.
• CSI NodePublish failure — the CSI node plugin errors during NodePublishVolume (driver bug, bad mount options).
• Read-only or stale mount — a remounted subPath becomes stale after the source configMap is updated.

How to Reproduce the Error

Mount a configMap that does not exist:

apiVersion: v1
kind: Pod
metadata:
  name: failedmount-demo
spec:
  containers:
    - name: app
      image: registry.k8s.io/pause:3.9
      volumeMounts:
        - name: cfg
          mountPath: /etc/app
  volumes:
    - name: cfg
      configMap:
        name: does-not-exist   # never created

kubectl apply -f failedmount-demo.yaml
kubectl describe pod failedmount-demo | grep -A4 Events

Warning  FailedMount  9s (x4 over 22s)  kubelet  MountVolume.SetUp failed for volume "cfg" : configmap "does-not-exist" not found

The pod stays ContainerCreating and the event repeats until the configMap is created.

Diagnostic Commands

# Read the specific FailedMount reason (the actual root cause)
kubectl describe pod <POD> | grep -A6 Events

# Map the failing volume name to its source (secret/configMap/csi/subPath)
kubectl get pod <POD> -o jsonpath='{range .spec.volumes[*]}{.name}{": "}{.configMap.name}{.secret.secretName}{.csi.driver}{"\n"}{end}'

# Does the referenced secret/configMap exist in THIS namespace?
kubectl get configmap <NAME> -n <NAMESPACE>
kubectl get secret <NAME> -n <NAMESPACE>

# If a specific key/subPath is referenced, confirm it exists
kubectl get configmap <NAME> -n <NAMESPACE> -o jsonpath='{.data}' | jq 'keys'

# Check the security context that governs file permissions
kubectl get pod <POD> -o jsonpath='{.spec.securityContext}{"\n"}'

# Node-side kubelet detail for CSI/subPath/permission failures
sudo journalctl -u kubelet --no-pager --since "10 min ago" | grep -i 'failedmount\|setup\|nodepublish\|subpath'

The describe pod event text is decisive — it names both the volume and the precise reason, so you rarely need to guess.

Step-by-Step Resolution

1. Read the reason after the volume name. The text following MountVolume.SetUp failed for volume "<name>" : is the actual cause — not found, subPath ... not found, or permission denied.

2. For configmap/secret not found, create it in the right namespace. Confirm existence and namespace, then create or move it:

kubectl get configmap app-config -n prod    # empty -> create it
kubectl create configmap app-config --from-file=app.conf -n prod

3. For a missing key, fix the source or mark it optional. Either add the key to the secret/configMap, correct the items reference, or set optional: true if absence is acceptable.

4. For subPath ... not found, align the subPath with real content. The subPath must reference a key/path that exists in the volume. Correct the subPath value or add the file to the source. Note: subPath mounts do not receive live updates when the configMap changes.

5. For permission denied, set the security context. Use fsGroup so the mounted files are group-readable by the container user:

spec:
  securityContext:
    fsGroup: 2000
    runAsUser: 1000

For SELinux nodes, set appropriate seLinuxOptions rather than disabling enforcement.

6. For CSI NodePublish errors, check the node plugin. Ensure the CSI node DaemonSet pod on that node is healthy and review its logs; bad mount options or a driver bug surface here.

7. Let it self-heal once fixed. The kubelet retries FailedMount on backoff. As soon as the missing object exists or the subPath is valid, the next retry mounts and the pod proceeds — no pod recreation needed for missing config objects.

Prevention and Best Practices

• Deploy secrets and configMaps before or alongside the workloads that mount them; ordering tools (Helm hooks, Argo sync waves, Kustomize) prevent the race that produces not found.
• Use optional: true on secret/configMap volumes whose absence the app can tolerate, so a missing object degrades gracefully instead of blocking startup.
• Avoid subPath for files you expect to update at runtime — subPath mounts are static and will not pick up configMap changes; mount the whole volume instead.
• Set fsGroup/runAsUser explicitly when running as non-root so mounted files are readable, avoiding permission denied.
• Keep config objects and their consumers in the same namespace; cross-namespace references silently fail to mount.
• Alert on FailedMount events so a missing object is caught at deploy time, not during a scaling event.

Related Errors

• MountVolume.SetUp failed — deeper dive on the CSI/persistent-volume side of setup failures.
• Unable to attach or mount volumes (timed out) — the aggregate timeout message this event rolls up into.
• PersistentVolumeClaim not bound — when there is no PV to mount at all.
• CreateContainerConfigError — a related “missing secret/configmap” failure at container config time.

Frequently Asked Questions

How is FailedMount different from “Unable to attach or mount volumes … timed out”? FailedMount with MountVolume.SetUp failed is the specific per-volume event that names the exact cause and fires immediately on each retry. The “timed out waiting for the condition” message is the kubelet’s aggregate summary after ~2 minutes and only lists volume names. Read the FailedMount event for the real reason.

My configMap exists but the mount still fails. Why? It is almost certainly in a different namespace than the pod, or a referenced items key does not exist within it. Volumes resolve config objects only within the pod’s own namespace. Verify with kubectl get configmap <name> -n <pod-namespace>.

Does the pod recover automatically after I create the missing secret? Yes. The kubelet retries FailedMount on an exponential backoff. Once the secret or configMap exists in the right namespace, the next retry mounts it and the pod continues to Running — no need to recreate the pod.

Why does my subPath mount not update when I change the configMap? subPath mounts are resolved once and are not kept in sync with the source object. If you need live updates, mount the entire volume at a directory instead of using subPath, and the kubelet will propagate changes.

What causes permission denied on a freshly mounted volume? The container’s user/group cannot read the mounted files. Set fsGroup (and runAsUser) in the pod securityContext so ownership matches, or adjust SELinux seLinuxOptions on enforcing nodes. Disabling security enforcement is not the right fix.