🚨 Kubernetes Error Guide: 'No preemption victims found for incoming pod' Pending Pods

Exact Error Message

A pod fails normal scheduling, the scheduler attempts preemption to make room, and that also fails — the pod stays Pending with a two-part FailedScheduling message:

Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  29s (x5 over 3m4s)  default-scheduler  0/5 nodes are available: 5 Insufficient cpu. preemption: 0/5 nodes are available: 5 No preemption victims found for incoming pod.

The message has two halves separated by preemption:. The first half (5 Insufficient cpu) is why ordinary scheduling failed. The second half (5 No preemption victims found for incoming pod) is why the fallback — evicting lower-priority pods — also could not place the pod.

What the Error Means

When a pod cannot be scheduled, the kube-scheduler may try preemption: if the pending pod has higher priority than some running pods, it can evict (preempt) them to free capacity. Preemption only happens for pods with a priorityClassName whose priority exceeds that of potential victims, and only when preemptionPolicy is PreemptLowerPriority (the default).

No preemption victims found for incoming pod means the scheduler ran the preemption logic and concluded that evicting any set of lower-priority pods on any node would still not make the pod schedulable — or there were no lower-priority pods to evict at all. Common reasons: the pending pod has equal-or-lower priority than everything running (nothing is a valid victim), evicting candidates still would not satisfy the pod’s requests/affinity, or PodDisruptionBudgets and preemptionPolicy: Never block eviction.

The first half of the message tells you the underlying constraint; preemption is just the failed rescue attempt. If the first half is Insufficient cpu, the real fix is capacity. If it is an affinity or taint failure, preemption was never going to help.

Common Causes

• Pod priority too low — the pending pod has no priorityClassName, or one whose value does not exceed running pods, so nothing qualifies as a victim.
• Victims would not free enough — even after evicting all lower-priority pods on a node, requests/affinity/taints still cannot be satisfied.
• All workloads at equal priority — everything uses the same class, so there is no “lower priority” to preempt.
• preemptionPolicy: Never — the priority class or pod explicitly forbids preemption, so the scheduler will not evict anyone.
• PodDisruptionBudget blocks eviction — preempting the only candidate would violate a PDB, so it is skipped.
• Underlying constraint is not capacity — the headline reason is a taint or affinity mismatch that eviction cannot resolve.
• System-critical victims — candidate victims are themselves high-priority/system pods that cannot be evicted.

How to Reproduce the Error

Fill a small cluster with same-priority pods, then submit a higher-request pod with no higher priority:

apiVersion: v1
kind: Pod
metadata:
  name: preempt-demo
spec:
  # no priorityClassName -> priority 0, same as the pods already filling the cluster
  containers:
    - name: app
      image: registry.k8s.io/pause:3.9
      resources:
        requests:
          cpu: "4"   # more than any node has free

kubectl apply -f preempt-demo.yaml
kubectl describe pod preempt-demo | grep -A4 Events

0/5 nodes are available: 5 Insufficient cpu. preemption: 0/5 nodes are available: 5 No preemption victims found for incoming pod.

Because the pod’s priority is not higher than the running pods, none are valid victims.

Diagnostic Commands

# Read both halves of the scheduling/preemption message
kubectl describe pod <POD> | grep -A6 Events

# What priority does the pending pod actually have?
kubectl get pod <POD> -o jsonpath='{.spec.priorityClassName} {.spec.priority}{"\n"}'

# List priority classes and their values
kubectl get priorityclasses -o custom-columns=NAME:.metadata.name,VALUE:.value,DEFAULT:.globalDefault,POLICY:.preemptionPolicy

# Inspect running pods' priorities on the candidate node (potential victims)
kubectl get pods -A --field-selector spec.nodeName=<NODE> \
  -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,PRIO:.spec.priority

# Check the pending pod's own preemptionPolicy
kubectl get pod <POD> -o jsonpath='{.spec.preemptionPolicy}{"\n"}'

# Are PodDisruptionBudgets protecting the candidates?
kubectl get pdb -A

Compare the pending pod’s priority against the running pods’ priorities on the node — if it is not strictly higher, there are simply no victims.

Step-by-Step Resolution

1. Read the first half first. Whatever precedes preemption: is the real constraint. If it is Insufficient cpu/memory, this is a capacity problem; if it is a taint or affinity reason, preemption was never applicable.

2. Confirm the pending pod’s priority. If it has no priorityClassName, its priority is 0 and it cannot preempt anything at priority 0. Assign an appropriate class:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
preemptionPolicy: PreemptLowerPriority
globalDefault: false
description: "Critical workloads that may preempt best-effort pods."

spec:
  priorityClassName: high-priority

3. Verify victims would actually help. Even with higher priority, if evicting lower-priority pods still leaves the node short on CPU/memory or violates affinity, no victim set works. In that case, add capacity rather than relying on preemption.

4. Check preemptionPolicy. If the pod or its class sets Never, the scheduler will not evict. Switch to PreemptLowerPriority if preemption is intended.

5. Account for PodDisruptionBudgets. A PDB that would be violated by eviction makes a candidate ineligible. Loosen the PDB or add capacity so eviction is unnecessary.

6. The durable fix is usually capacity. Preemption is a stopgap. Run a cluster autoscaler so Insufficient cpu adds a node instead of fighting over preemption victims.

7. Avoid a flat priority landscape. Define a small tier of priority classes (best-effort, normal, critical) so the scheduler has lower-priority pods it can actually preempt.

Prevention and Best Practices

• Design a deliberate priority tier so critical pods can preempt best-effort ones; a cluster where everything is the same priority can never preempt.
• Treat preemption as a safety valve, not a capacity strategy — pair it with a cluster autoscaler so the common case adds nodes.
• Set preemptionPolicy: Never on batch/best-effort priority classes you never want to disturb others, and keep PreemptLowerPriority for genuinely critical tiers.
• Keep PodDisruptionBudgets realistic; overly strict PDBs block both preemption and voluntary disruptions.
• Right-size requests so high-priority pods do not need to evict large victim sets just to fit.
• Alert on pods Pending with a No preemption victims message — it signals both a capacity shortfall and a priority-design gap.

Related Errors

• FailedScheduling — the umbrella scheduling page; the first half of this message routes here.
• Insufficient cpu / memory — the most common underlying constraint behind failed preemption.
• Pod affinity rules not matched — a constraint preemption cannot resolve.
• Untolerated taint / node affinity — another non-capacity reason preemption fails.

Frequently Asked Questions

Why did the scheduler even try to preempt? Normal scheduling failed, so the scheduler falls back to preemption to see whether evicting lower-priority pods would help. The No preemption victims found clause is the result of that evaluation — it tried and found no useful eviction.

My pod has a high priority class but still finds no victims. Why? Either the running pods on the candidate nodes are at equal-or-higher priority (not valid victims), evicting the eligible ones still would not satisfy your requests/affinity, or a PodDisruptionBudget protects them. Inspect per-node pod priorities and PDBs.

Should I rely on preemption to keep critical pods scheduled? No. Preemption is a last resort that disrupts running workloads. The reliable pattern is right-sized requests plus a cluster autoscaler, with priority classes as a safety net for genuine contention.

What does preemptionPolicy: Never do? It makes a pod (via its priority class) eligible to be scheduled with high priority but never to evict others. Useful for high-priority-but-polite workloads. If set, the scheduler will report no victims because it is not allowed to look for any.

Is the first or second half of the message more important? The first half — it names the real constraint. Preemption is only the failed rescue. Fix the underlying Insufficient cpu, taint, or affinity issue and the preemption clause becomes moot.