🚨 Kubernetes Error Guide: 'node(s) had volume node affinity conflict' — Fix Pod Scheduling

Overview

volume node affinity conflict is a scheduling failure: the pod needs a PersistentVolume that is pinned to a specific topology (usually one availability zone or one node), and no schedulable node in that topology can also satisfy the pod’s other constraints. The scheduler cannot place the pod on any node that can both attach the volume and meet its requirements, so the pod stays Pending.

You will see it in the pod’s events:

Warning  FailedScheduling  0/6 nodes are available: 3 node(s) had volume node affinity conflict,
3 node(s) didn't match Pod's node affinity/selector. preemption: 0/6 nodes are available.

The PV carries a nodeAffinity block (set at provisioning time by the CSI driver or the admin) that says “this volume lives in zone us-east-1a.” If every node in us-east-1a is cordoned, full, tainted, or excluded by the pod’s own affinity, the volume becomes unreachable and the pod cannot schedule. This is most common with zonal block storage (EBS, GCE PD, Azure Disk) and local PVs.

Symptoms

• Pod is stuck Pending; kubectl describe pod shows FailedScheduling mentioning volume node affinity conflict.
• A previously-running StatefulSet pod will not reschedule after a node drain or zone outage.
• The count of conflicting nodes matches the nodes outside the volume’s zone.
• The PV shows a nodeAffinity requirement pinned to one zone or one node.

kubectl get pod postgres-0 -o wide

NAME         READY   STATUS    RESTARTS   AGE   IP    NODE     NOMINATED NODE
postgres-0   0/1     Pending   0          6m    <none> <none>  <none>

kubectl describe pod postgres-0 | grep -A3 Events

Events:
  Type     Reason            Age                From               Message
  Warning  FailedScheduling  5m (x8 over 6m)    default-scheduler  0/6 nodes are available:
           3 node(s) had volume node affinity conflict, 3 node(s) didn't match node selector.

Common Root Causes

1. Zonal PV, but the pod’s zone has no schedulable nodes

The PV is bound to, say, us-east-1a, but every node in 1a is cordoned, drained, or was removed by the autoscaler. The volume can only attach in 1a, so the pod cannot run anywhere.

kubectl get pv $(kubectl get pvc data-postgres-0 -o jsonpath='{.spec.volumeName}') \
  -o jsonpath='{.spec.nodeAffinity}{"\n"}'

{"required":{"nodeSelectorTerms":[{"matchExpressions":[
  {"key":"topology.kubernetes.io/zone","operator":"In","values":["us-east-1a"]}]}]}}

kubectl get nodes -l topology.kubernetes.io/zone=us-east-1a

No resources found

The zone has zero nodes. Uncordon/scale a node in 1a, or restore capacity there.

2. Pod affinity or nodeSelector fights the volume’s zone

The pod (or its controller) has a nodeSelector/nodeAffinity that pins it to a different zone than the PV. Each is satisfiable alone, but not together.

kubectl get pod postgres-0 -o jsonpath='{.spec.nodeSelector}{"\n"}'

{"topology.kubernetes.io/zone":"us-east-1b"}

The pod wants 1b; the PV lives in 1a. Remove the conflicting selector or provision the volume in the pod’s target zone.

3. StorageClass without WaitForFirstConsumer provisioned the PV in the wrong zone

With volumeBindingMode: Immediate, the volume is provisioned before the pod is scheduled, so the CSI driver picks a zone blindly — often not where the pod can run. WaitForFirstConsumer defers binding until the scheduler picks a node, avoiding the mismatch.

kubectl get storageclass gp3 -o jsonpath='{.volumeBindingMode}{"\n"}'

Immediate

4. Local PV whose node is gone or cordoned

A local PersistentVolume is pinned to exactly one node. If that node is drained, tainted, or deleted, the data is unreachable and any pod bound to it cannot schedule.

kubectl get pv local-pv-abc -o jsonpath='{.spec.nodeAffinity}{"\n"}'
kubectl get node worker-3 -o jsonpath='{.spec.taints}{"\n"}'

Diagnostic Workflow

Step 1: Confirm the pod is Pending on this exact reason

kubectl describe pod <POD> | grep -A6 Events

Look for volume node affinity conflict specifically — a generic FailedScheduling about CPU/taints is a different problem.

Step 2: Find the PVC → PV → topology chain

PVC=$(kubectl get pod <POD> -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim)].persistentVolumeClaim.claimName}')
PV=$(kubectl get pvc "$PVC" -o jsonpath='{.spec.volumeName}')
kubectl get pv "$PV" -o jsonpath='{.spec.nodeAffinity}{"\n"}'

This prints the exact zone/node the volume is pinned to.

Step 3: Check whether any schedulable node matches that topology

# Substitute the zone from Step 2
kubectl get nodes -l topology.kubernetes.io/zone=<ZONE> -o wide
kubectl get nodes -l topology.kubernetes.io/zone=<ZONE> \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.unschedulable}{"\n"}{end}'

Zero nodes, or all unschedulable: true, is the conflict.

Step 4: Compare against the pod’s own placement constraints

kubectl get pod <POD> -o jsonpath='{.spec.nodeSelector}{"\n"}'
kubectl get pod <POD> -o jsonpath='{.spec.affinity}{"\n"}'

If the pod is pinned to a different zone than the PV, that is the conflict.

Step 5: Check the StorageClass binding mode

SC=$(kubectl get pvc "$PVC" -o jsonpath='{.spec.storageClassName}')
kubectl get storageclass "$SC" -o jsonpath='{.volumeBindingMode}{"\n"}'

Immediate on a multi-zone cluster is a recurring cause; WaitForFirstConsumer is the fix for new volumes.

Example Root Cause Analysis

A StatefulSet pod redis-0 won’t come back after routine node maintenance.

kubectl describe pod redis-0 | grep -A4 Events

Warning  FailedScheduling  2m (x15)  default-scheduler  0/9 nodes are available:
6 node(s) didn't match pod affinity, 3 node(s) had volume node affinity conflict.

Trace the volume’s zone:

kubectl get pv $(kubectl get pvc data-redis-0 -o jsonpath='{.spec.volumeName}') \
  -o jsonpath='{.spec.nodeAffinity}{"\n"}'

...topology.kubernetes.io/zone In [us-west-2c]...

Check nodes in us-west-2c:

kubectl get nodes -l topology.kubernetes.io/zone=us-west-2c \
  -o custom-columns=NAME:.metadata.name,SCHED:.spec.unschedulable

NAME              SCHED
ip-10-2-3-4       true

The only node in 2c was cordoned during maintenance and never uncordoned. The zonal EBS volume can attach only in 2c, so redis-0 is stuck. Fix:

kubectl uncordon ip-10-2-3-4

Within a few seconds the scheduler places redis-0 on that node, the volume attaches, and the pod goes Running. The durable fix is to ensure each zone that holds stateful volumes always retains schedulable capacity, and to run enough replicas across zones that one cordoned node is not a single point of failure.

Prevention Best Practices

• Use volumeBindingMode: WaitForFirstConsumer on multi-zone clusters so volumes are provisioned in the zone where the pod actually schedules — this eliminates the most common cause for new PVCs.
• Keep schedulable capacity in every zone that holds zonal or local PVs; never drain the last node of a zone that pins stateful data.
• For StatefulSets, spread replicas across zones (topology spread constraints) so a single zone’s outage cannot strand the whole set.
• Avoid pinning pods with nodeSelector to a zone different from where their volumes live; let the volume topology drive placement.
• For local PVs, treat the owning node as part of the data’s durability story — its loss means the data is gone, not just delayed.
• Watch autoscaler behavior: a scale-in that removes the last node in a zone can strand volumes there. See more in Kubernetes & Helm guides.

Quick Command Reference

# Confirm the reason
kubectl describe pod <POD> | grep -A6 Events

# PVC -> PV -> pinned topology
PVC=$(kubectl get pod <POD> -o jsonpath='{.spec.volumes[?(@.persistentVolumeClaim)].persistentVolumeClaim.claimName}')
PV=$(kubectl get pvc "$PVC" -o jsonpath='{.spec.volumeName}')
kubectl get pv "$PV" -o jsonpath='{.spec.nodeAffinity}{"\n"}'

# Schedulable nodes in the volume's zone
kubectl get nodes -l topology.kubernetes.io/zone=<ZONE> \
  -o custom-columns=NAME:.metadata.name,SCHED:.spec.unschedulable

# Pod's own placement constraints
kubectl get pod <POD> -o jsonpath='{.spec.nodeSelector} {.spec.affinity}{"\n"}'

# StorageClass binding mode
kubectl get storageclass <SC> -o jsonpath='{.volumeBindingMode}{"\n"}'

# Common fixes
kubectl uncordon <NODE>                       # restore capacity in the zone
# or provision new volumes with WaitForFirstConsumer StorageClass

Conclusion

node(s) had volume node affinity conflict means the scheduler cannot find a node that both satisfies the pod’s constraints and can attach a topology-pinned volume. The usual root causes:

1. The PV’s zone has no schedulable nodes (cordoned, drained, or scaled away).
2. The pod’s own nodeSelector/affinity pins it to a different zone than the volume.
3. volumeBindingMode: Immediate provisioned the PV in the wrong zone before scheduling.
4. A local PV’s owning node is gone, tainted, or unschedulable.

Trace PVC → PV → nodeAffinity to learn the volume’s zone, then check for schedulable nodes there. The lasting fix is WaitForFirstConsumer plus keeping capacity in every zone that holds stateful data. For ad-hoc triage, the free incident assistant can turn a scheduling describe dump into the likely root cause.