🚨 Kubernetes Error Guide: '5 node(s) didn't match pod affinity rules' Pending Pods

Exact Error Message

A pod with inter-pod affinity rules cannot be placed, so it stays Pending and the scheduler records a FailedScheduling event naming the affinity predicate:

Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  34s (x4 over 2m1s)  default-scheduler  0/5 nodes are available: 5 node(s) didn't match pod affinity rules. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.

The anti-affinity variant reads:

0/5 nodes are available: 5 node(s) didn't match pod anti-affinity rules.

5 node(s) didn't match pod affinity rules means every candidate node failed the requiredDuringSchedulingIgnoredDuringExecution inter-pod affinity check. Note Preemption is not helpful — evicting lower-priority pods cannot fix an affinity mismatch, so the pod simply waits.

What the Error Means

Inter-pod affinity schedules a pod relative to other pods, not to node labels. podAffinity says “place me on a node in the same topology domain as pods matching this label selector”; podAntiAffinity says “keep me away from them.” The topologyKey defines the domain — kubernetes.io/hostname means per-node, topology.kubernetes.io/zone means per-zone.

When the scheduler evaluates a required affinity rule, it looks for nodes whose topology domain already contains (for affinity) or is free of (for anti-affinity) pods matching the selector. didn't match pod affinity rules means no node’s domain satisfied the requirement — for affinity, no domain held a matching “anchor” pod; for anti-affinity, every domain already held a conflicting pod.

Because the constraint is about pod placement rather than capacity, the cluster can be wide open on CPU and memory and the pod will still be unschedulable. Preemption does not help, which is why the message explicitly says so.

Common Causes

• No anchor pod exists yet — a required podAffinity references a label selector that matches zero running pods, so there is no domain to join (classic chicken-and-egg on first deploy).
• Anchor pods only in domains without capacity — matching pods exist, but their nodes/zones cannot fit the new pod.
• Anti-affinity saturation — requiredDuringScheduling anti-affinity with topologyKey: hostname and more replicas than nodes; once every node has one, the rest cannot place.
• Wrong or missing topologyKey label — nodes lack the label named in topologyKey (e.g. zone labels absent), so no node forms a valid domain.
• Label selector typo — the affinity labelSelector does not match the intended pods’ actual labels.
• Wrong namespace scope — affinity defaults to the pod’s own namespace; the anchor pods live elsewhere and are invisible to the rule.
• Over-strict required rules — using requiredDuringScheduling where preferred was intended.

How to Reproduce the Error

Require co-location with a pod label that does not exist:

apiVersion: v1
kind: Pod
metadata:
  name: affinity-demo
  labels: { app: web }
spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels: { app: cache }   # no 'cache' pods running
          topologyKey: kubernetes.io/hostname
  containers:
    - name: app
      image: registry.k8s.io/pause:3.9

kubectl apply -f affinity-demo.yaml
kubectl get pod affinity-demo

NAME            READY   STATUS    RESTARTS   AGE
affinity-demo   0/1     Pending   0          15s

kubectl describe pod affinity-demo reports 5 node(s) didn't match pod affinity rules because no node’s hostname domain contains an app=cache pod.

Diagnostic Commands

# Read the affinity failure in the scheduling event
kubectl describe pod <POD> | grep -A6 Events

# Dump the pod's affinity rules and the selectors/topologyKeys they use
kubectl get pod <POD> -o jsonpath='{.spec.affinity}' | jq

# Are there any pods matching the affinity selector, and where?
kubectl get pods -A -l app=cache -o wide

# Do nodes carry the label used as topologyKey?
kubectl get nodes -L kubernetes.io/hostname -L topology.kubernetes.io/zone

# Count nodes vs replicas for anti-affinity saturation
kubectl get nodes --no-headers | wc -l
kubectl get deploy <DEPLOY> -o jsonpath='{.spec.replicas}{"\n"}'

# Confirm the namespace the affinity actually searches
kubectl get pod <POD> -o jsonpath='{.metadata.namespace}{"\n"}'

Cross-check three things: do matching anchor pods exist, do nodes carry the topologyKey label, and (for anti-affinity) is replica count ≤ domain count.

Step-by-Step Resolution

1. Identify affinity vs anti-affinity. The event says which. Affinity = “needs a neighbor it cannot find.” Anti-affinity = “needs separation it cannot get.”

2. For affinity with no anchor, deploy or fix the selector. Ensure pods matching the labelSelector actually run, or correct a label typo:

kubectl get pods -A -l app=cache -o wide   # empty -> nothing to co-locate with

3. Verify the topologyKey label exists on nodes. If topologyKey: topology.kubernetes.io/zone but nodes are unlabeled, no domain forms. Label nodes or switch to kubernetes.io/hostname which every node has.

4. For anti-affinity saturation, add nodes or relax the rule. If replicas exceed nodes under topologyKey: hostname, either scale the node pool or change requiredDuringScheduling to preferredDuringScheduling:

podAntiAffinity:
  preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        labelSelector:
          matchLabels: { app: web }
        topologyKey: kubernetes.io/hostname

5. Fix namespace scope. If anchor pods live in another namespace, add namespaces: or namespaceSelector: to the affinity term so the scheduler can see them.

6. Re-evaluate after each change. Affinity constraints interact; clearing one mismatch can reveal a capacity limit next. Re-read the event after each fix.

7. Consider topologySpreadConstraints. For “spread evenly but do not hard-fail,” topologySpreadConstraints with whenUnsatisfiable: ScheduleAnyway is often a better fit than required anti-affinity.

Prevention and Best Practices

• Prefer preferredDuringSchedulingIgnoredDuringExecution unless co-location or separation is truly mandatory; required rules turn ordinary capacity pressure into hard Pending.
• For “spread my replicas,” use topologySpreadConstraints instead of hostname anti-affinity — it degrades gracefully and is purpose-built for the job.
• Keep requiredDuringScheduling anti-affinity replica counts ≤ the number of topology domains, and scale the node pool alongside replicas.
• Ensure every node carries the labels your topologyKeys reference; missing zone labels silently break domain formation.
• Validate affinity labelSelectors against the real pod labels in CI to catch typos before they reach the scheduler.
• Be explicit about namespace scope when the anchor pods live outside the pod’s own namespace.

Related Errors

• FailedScheduling — the umbrella scheduling-failure page and how to read the breakdown.
• No preemption victims found — why preemption does not rescue affinity mismatches.
• Insufficient cpu / memory — the capacity cause that often hides behind affinity domains.
• Untolerated taint / node affinity — node-level placement constraints.

Frequently Asked Questions

What is the difference between pod affinity and node affinity? Node affinity matches node labels (“run me on SSD nodes”). Pod affinity matches other pods within a topology domain (“run me next to the cache”). This error is the inter-pod kind — it depends on where other pods are, not on node labels alone.

Why does the message say preemption is not helpful? Preemption evicts lower-priority pods to free capacity. Affinity failures are not about capacity — evicting pods cannot create a missing anchor pod or undo anti-affinity saturation. So the scheduler reports preemption as useless here.

My first replica schedules but the rest go Pending — why? Classic required anti-affinity with topologyKey: hostname and more replicas than nodes. The first replica claims a node; the others cannot find a node without a same-app pod. Add nodes or switch to a preferred rule or topology spread.

What does topologyKey actually control? It names the node label whose value defines a “domain.” kubernetes.io/hostname makes each node its own domain (strict per-node); topology.kubernetes.io/zone groups nodes by zone. The affinity/anti-affinity check is evaluated per domain.

Does pod affinity look across all namespaces? No. By default it only considers pods in the same namespace as the pod being scheduled. Use the namespaces list or namespaceSelector field if your anchor pods live elsewhere.