🚨 Kubernetes Error Guide: 'Unable to attach or mount volumes: timed out waiting for the condition'

Exact Error Message

A pod schedules successfully but never reaches Running because the kubelet cannot get its volumes ready within the timeout. After roughly two minutes the kubelet emits a summary event listing what is still unmounted and unattached:

Events:
  Type     Reason       Age                  From     Message
  ----     ------       ----                 ----     -------
  Warning  FailedMount  2m13s (x2 over 4m)   kubelet  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data kube-api-access-7q4mz]: timed out waiting for the condition

The pod sits in ContainerCreating the whole time:

NAME       READY   STATUS              RESTARTS   AGE
api-0      0/1     ContainerCreating   0          4m12s

This is the kubelet’s summary message after the volume manager’s two-minute deadline elapsed. unmounted volumes=[...] lists volumes not yet mounted into the pod; unattached volumes=[...] lists volumes the kubelet is still waiting to be attached to the node.

What the Error Means

Bringing a volume online for a pod is a multi-stage pipeline: provision (create the disk) → attach (bind it to the node) → mount/NodeStage + NodePublish (format if needed and mount into the pod). The kubelet’s volume manager waits up to ~2 minutes for all of a pod’s volumes to complete this pipeline. If any volume is not ready when the deadline hits, the kubelet aborts the attempt and reports Unable to attach or mount volumes ... timed out waiting for the condition.

The two lists tell you where in the pipeline things are stuck:

• A volume in both unmounted and unattached is stuck early — it has not even attached to the node (CSI attach failing, disk in another zone, volume still attached elsewhere).
• A volume in unmounted but not unattached attached fine but failed to mount (filesystem, NodeStage/NodePublish, subPath, or permissions).

This is the aggregate, timeout-level message. The specific cause usually appears in an earlier, more detailed FailedMount/FailedAttachVolume event — this guide helps you map the summary to that root cause. For the per-mount detail, see the FailedMount event guide.

Common Causes

• CSI attach failure — the CSI controller cannot attach the disk (cloud API throttling, IAM permissions, driver crash).
• Multi-attach on RWO — the volume is still attached to another node because the old pod has not fully terminated.
• Zone/topology mismatch — the PV lives in one availability zone and the pod was scheduled onto a node in another.
• PVC not bound — the claim never bound to a PV, so there is nothing to attach.
• CSI node plugin not running — the node’s CSI DaemonSet pod is missing or unhealthy, so NodeStage/NodePublish never runs.
• Filesystem/format issues — the volume cannot be formatted or fsck stalls during mount.
• Cloud-side disk problems — the underlying disk is deleting, in an error state, or quota-limited.
• Secret/configmap volume missing — a referenced secret or configmap does not exist, so that projected volume never mounts.

How to Reproduce the Error

Bind a pod to a PVC whose PV is pinned to a zone with no schedulable node, or simply reference a not-yet-bound PVC:

apiVersion: v1
kind: Pod
metadata:
  name: mount-timeout-demo
spec:
  containers:
    - name: app
      image: registry.k8s.io/pause:3.9
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: stuck-pvc   # bound to a PV in a zone with no node

kubectl apply -f mount-timeout-demo.yaml
kubectl get pod mount-timeout-demo

NAME                 READY   STATUS              RESTARTS   AGE
mount-timeout-demo   0/1     ContainerCreating   0          3m

After ~2 minutes, kubectl describe pod shows Unable to attach or mount volumes ... timed out waiting for the condition.

Diagnostic Commands

# Read the summary plus any earlier, more specific mount/attach events
kubectl describe pod <POD> | grep -A12 Events

# Map the pod's volumes to their PVCs and PVs
kubectl get pod <POD> -o jsonpath='{range .spec.volumes[*]}{.name}{" -> "}{.persistentVolumeClaim.claimName}{"\n"}{end}'
kubectl get pvc -n <NAMESPACE>
kubectl get pv <PV>

# Is the volume attached, and to which node? (the early vs late check)
kubectl get volumeattachment | grep <PV>

# Compare the PV's zone topology against the node's zone
kubectl get pv <PV> -o jsonpath='{.spec.nodeAffinity}' | jq
kubectl get node <NODE> -L topology.kubernetes.io/zone

# Is the CSI node plugin healthy on this node?
kubectl get pods -n kube-system -o wide | grep -i csi | grep <NODE>

# Node-side kubelet view of the stuck mount
sudo journalctl -u kubelet --no-pager --since "10 min ago" | grep -i 'mount\|attach\|csi'

The presence or absence of a VolumeAttachment for the PV is the fastest way to tell “stuck at attach” from “stuck at mount.”

Step-by-Step Resolution

1. Read the two lists. Determine whether the volume is stuck at attach (in both lists) or at mount (only unmounted). This decides which half of the pipeline to investigate.

2. If unattached, check the VolumeAttachment and zone. Look for a VolumeAttachment for the PV and confirm the node’s zone matches the PV’s nodeAffinity. A zone mismatch means the pod must be rescheduled into the PV’s zone:

kubectl get volumeattachment | grep <PV>
kubectl get pv <PV> -o jsonpath='{.spec.nodeAffinity}' | jq

3. Clear multi-attach. For an RWO volume still attached to an old node, confirm the previous pod is fully gone; force-delete a stuck terminating pod so the volume detaches. See the multi-attach guide.

4. If attached but unmounted, inspect the CSI node plugin. Ensure the CSI DaemonSet pod on that node is Running; restart it if crashed. Check the detailed FailedMount event for NodeStage/NodePublish, subPath, or filesystem errors.

5. Fix PVC binding. If the PVC is Pending, resolve binding first — no PV means nothing to attach. See the PVC not bound guide.

6. Check cloud-side disk state and quotas. Throttled cloud APIs or a disk in an error state stall attach. Review CSI controller logs (kubectl logs -n kube-system <csi-controller>) for the cloud error.

7. Use WaitForFirstConsumer to prevent zone mismatches. A volumeBindingMode: WaitForFirstConsumer StorageClass defers provisioning until the pod is scheduled, so the disk is created in the right zone.

Prevention and Best Practices

• Use volumeBindingMode: WaitForFirstConsumer StorageClasses so volume topology matches the scheduled node — the single biggest preventer of attach timeouts.
• Monitor the CSI controller and node DaemonSet health; a crashed node plugin silently stalls every mount on that node.
• For RWO volumes, ensure clean pod termination and consider StatefulSet ordering so a replacement does not race the detach of its predecessor.
• Watch cloud-provider attach quotas and API throttling; spreading attach operations avoids deadline-blowing backpressure.
• Alert on pods stuck in ContainerCreating beyond a couple of minutes — this timeout will not self-resolve without intervention or a state change.
• Keep PVCs bound proactively and avoid manual PV/zone pinning unless required.

Related Errors

• MountVolume.SetUp failed — the per-volume mount failure behind the unmounted list.
• FailedAttachVolume / Multi-Attach — the attach failure behind the unattached list.
• PersistentVolumeClaim not bound — when there is no PV to attach at all.
• FailedScheduling — volume node-affinity conflicts that block scheduling earlier.

Frequently Asked Questions

What is the difference between the unmounted and unattached lists? unattached volumes have not yet been attached to the node — the failure is early (attach/topology/multi-attach). unmounted volumes may be attached but not yet mounted into the pod — the failure is later (CSI node mount, subPath, filesystem). A volume in both lists is stuck at attach.

Why does it take about two minutes to appear? The kubelet’s volume manager waits a fixed deadline (~2 minutes) for the whole attach-and-mount pipeline to complete before declaring failure. The message is the timeout summary, not the first sign of trouble — earlier events hold the specific cause.

The pod was rescheduled to a new node and now fails to mount. Why? Most likely a zone/topology mismatch: the PV is pinned to one availability zone and the pod landed on a node in another, or the old node still holds an RWO attachment. Check the PV nodeAffinity and any lingering VolumeAttachment.

Is this the same as MountVolume.SetUp failed? They are related layers. Unable to attach or mount volumes ... timed out is the kubelet’s aggregate timeout message; MountVolume.SetUp failed is the specific per-volume mount error that often causes it. Read the detailed event to find the real root cause.

How do I prevent zone-mismatch attach timeouts? Use a WaitForFirstConsumer StorageClass so the volume is provisioned only after the pod is scheduled, guaranteeing the disk is created in the node’s zone. Avoid manually pinning PVs to zones unless you also constrain pod scheduling.