🚨 Kubernetes Error Guide: 'timed out waiting for the condition' Attach/Mount Timeout

Exact Error Message

A pod is stuck in ContainerCreating and kubectl describe pod reports a FailedMount event whose only detail is a generic timeout:

Events:
  Type     Reason       Age                 From       Message
  ----     ------       ----                ----       -------
  Warning  FailedMount  3m2s (x2 over 7m)   kubelet    Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data]: timed out waiting for the condition
  Warning  FailedMount  58s                 kubelet    MountVolume.WaitForAttach failed for volume "pvc-3b1f..." : timed out waiting for the condition

timed out waiting for the condition is deliberately vague. It is the kubelet’s volume manager giving up after waiting (about 2 minutes per cycle) for some sub-operation to complete. The message tells you that something timed out, not what.

What the Error Means

The kubelet drives volumes through stages — attach, WaitForAttach, mount/stage, then SetUp — and each stage polls a condition until it is true or a deadline passes. When the deadline passes, the polling helper returns the literal Go error string timed out waiting for the condition, and the kubelet bubbles it up as a FailedMount.

So this is a symptom, not a root cause. The real failure is one layer down: the CSI attach call is hanging, the device never appeared on the node, a network filesystem is unreachable, or a previous mount is stuck. Your job is to find which stage timed out and read the error underneath it. The clause unmounted volumes=[...] versus unattached volumes=[...] tells you whether the volume even attached.

Common Causes

• Attach never completed — the CSI/cloud attach is slow or throttled (often paired with rpc error: DeadlineExceeded).
• Device never appeared — WaitForAttach polls for a block device that the driver never surfaced on the node.
• Stuck/dead node holding the volume — an RWO volume cannot detach from an unreachable node, so attach waits forever.
• Unreachable network storage — NFS/SMB/Ceph endpoint is down or blocked by a firewall, so mount hangs.
• CSI node plugin missing or crash-looping — no driver on the node to perform NodeStage/NodePublish.
• fsGroup permission walk on huge volumes — applying ownership recursively to millions of files exceeds the mount window.

How to Reproduce the Error

Point a pod at an NFS server that does not exist; the mount will hang until the kubelet times out:

apiVersion: v1
kind: Pod
metadata:
  name: bad-nfs
spec:
  containers:
    - name: app
      image: registry.k8s.io/pause:3.9
      volumeMounts:
        - { name: data, mountPath: /data }
  volumes:
    - name: data
      nfs:
        server: 10.0.0.250   # no NFS server here
        path: /exports/data

kubectl apply -f bad-nfs.yaml
kubectl describe pod bad-nfs | grep -A6 Events

After roughly two minutes the pod shows FailedMount ... timed out waiting for the condition.

Diagnostic Commands

# Note whether the volume is "unattached" vs only "unmounted"
kubectl describe pod <POD> | grep -A10 Events

# Look for a more specific error from the attach controller
kubectl get events --field-selector involvedObject.name=<POD> --sort-by=.lastTimestamp

# Is the volume stuck attached to another (possibly dead) node?
kubectl get volumeattachment | grep <PV-NAME>

# Is a CSI node plugin present and healthy on the pod's node?
kubectl get pods -n kube-system -o wide | grep <NODE>

# Kubelet's own view of the stuck mount (read-only)
journalctl -u kubelet --since "10 min ago" | grep -i 'mount\|attach\|<PV-NAME>'

The goal of every command here is the same: replace the generic timeout with the specific error the stage actually hit.

Step-by-Step Resolution

1. Read attached vs. unattached. In unmounted volumes=[data], unattached volumes=[data], if the volume is listed as unattached, the failure is at the attach stage — go to step 2. If it attached but is only unmounted, the failure is at mount/stage — go to step 4.

2. Investigate the attach stage. Check for a sibling FailedAttachVolume event and look at VolumeAttachment objects. A stale attachment to a dead node is the classic cause; confirm the old node is gone and clear it:

kubectl describe volumeattachment <VA-NAME>
# only after confirming the old node/volume is detached cloud-side
kubectl delete volumeattachment <VA-NAME>

If you also see rpc error: DeadlineExceeded, treat it as a CSI attach timeout.

3. Confirm the CSI driver is healthy on the node. No node plugin means nothing performs the attach/mount. Ensure the driver’s DaemonSet pod is Running on the pod’s node.

4. Investigate the mount stage. For network storage, verify the endpoint is reachable and the export path exists. For block volumes, check that the device appeared. Read the node plugin log for the specific NodeStageVolume error:

kubectl logs -n kube-system <CSI-NODE-POD> -c <DRIVER> --tail=80

5. Check for fsGroup-on-large-volume stalls. If the volume is huge and the pod sets securityContext.fsGroup, the recursive ownership change can exceed the window. Set fsGroupChangePolicy: OnRootMismatch to skip already-correct trees.

6. Re-check after the fix. Once the underlying stage can complete, the kubelet’s next retry mounts cleanly:

kubectl get pod <POD> -o wide -w

Prevention and Best Practices

• Treat timed out waiting for the condition as a pointer, never a root cause — always find the stage-specific error before changing anything.
• Health-check network storage endpoints and alert when they go unreachable, since mounts will silently hang.
• Replace unreachable nodes quickly so RWO volumes can detach and re-attach elsewhere.
• Use fsGroupChangePolicy: OnRootMismatch for large volumes to avoid permission-walk timeouts.
• Keep CSI driver DaemonSets healthy on every node and alert on restarts. More patterns in the Kubernetes & Helm guides.

Related Errors

• rpc error: DeadlineExceeded during CSI attach/mount — the specific attach timeout this often hides.
• FailedAttachVolume Multi-Attach — RWO volume held by another node.
• MountVolume.SetUp failed — a more specific mount failure.
• PersistentVolumeClaim not bound — when the volume never provisioned in the first place.

Frequently Asked Questions

Why is the message so generic? It is a low-level Go polling helper (wait.Poll) returning its default error when a condition does not become true in time. Many different operations use that helper, so the same string appears for attach, mount, and detach timeouts alike.

How long does the kubelet wait before reporting it? Roughly two minutes per mount cycle, then it retries. That is why you often see the event repeat with (x2 over 7m) style counts — each cycle that fails adds another occurrence.

The volume eventually mounted on its own. What happened? A transient slowness (cloud throttling, a slow detach) resolved and a later retry succeeded. The timeout only describes a single failed attempt, not a permanent state.

unattached volumes vs unmounted volumes — what’s the difference? unattached means the volume never attached to the node, so the problem is at the attach/CSI controller layer. unmounted means it attached but did not mount, pointing at node-side staging, filesystem, or network-storage reachability.

Could this be a node problem rather than a storage problem? Yes. A NotReady or partially failed node can make every volume operation time out. Check node status and the kubelet/CSI node plugin health before assuming the volume itself is at fault.