🚨 GCP Error Guide: 'Cannot evict pod as it would violate the pod's disruption budget' — Fix GKE Drain

Overview

GKE (and any Kubernetes cluster) blocks a pod eviction when removing that pod would push a workload below the minimum availability declared in its PodDisruptionBudget (PDB). You see it when draining a node manually, and — more painfully — when a GKE node pool upgrade or maintenance stalls because the autoscaler or upgrade controller can’t drain a node:

error when evicting pods/"payments-api-7c9f8d6b5-2xk4p" -n prod:
 Cannot evict pod as it would violate the pod's disruption budget.

During an automated node pool upgrade the same condition surfaces in the operation status and Cloud Logging rather than on your terminal:

evicting pod prod/payments-api-7c9f8d6b5-2xk4p failed: the eviction would
 violate the disruption budget for the pod. Retrying...

Symptoms

• kubectl drain <node> hangs indefinitely, repeatedly logging the disruption-budget message.
• A GKE node pool upgrade sits in RUNNING for a very long time or the operation eventually times out.
• Cluster Autoscaler cannot scale down a node; autoscaler events cite a PDB blocking eviction.
• kubectl get pdb -A shows a PDB with ALLOWED DISRUPTIONS of 0.
• Nodes are stuck cordoned (SchedulingDisabled) because the drain never completes.

Common Root Causes

• ALLOWED DISRUPTIONS is zero — the workload currently has exactly minAvailable healthy pods, so evicting even one would breach the budget. Often the deployment is running fewer replicas than the PDB assumes.
• Unhealthy or unready replicas — pods exist but aren’t Ready, so the PDB counts fewer healthy pods than the replica count suggests and refuses any disruption.
• A single-replica deployment with minAvailable: 1 — mathematically impossible to disrupt; the drain can never succeed without a config change.
• maxUnavailable: 0 — a PDB expressed as maxUnavailable: 0 forbids all voluntary disruptions.
• Stuck or misconfigured pods — pods in CrashLoopBackOff/Pending never become Ready, permanently pinning ALLOWED DISRUPTIONS at 0.
• Orphaned PDBs — a PDB whose selector matches pods from an old or scaled-down workload, blocking drains for pods that shouldn’t be protected.

Diagnostic Workflow

Start by finding which PDB is blocking and how many disruptions it currently allows:

kubectl get pdb -A \
  -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,MIN:.spec.minAvailable,MAX:.spec.maxUnavailable,ALLOWED:.status.disruptionsAllowed

kubectl describe pdb payments-api -n prod

Compare the PDB’s expectation against the workload’s actual healthy replica count:

kubectl get deploy payments-api -n prod \
  -o custom-columns=NAME:.metadata.name,DESIRED:.spec.replicas,READY:.status.readyReplicas

kubectl get pods -n prod -l app=payments-api -o wide

If pods exist but aren’t Ready, find out why — an unready pod is what pins the budget at zero:

kubectl describe pod -n prod -l app=payments-api | grep -A5 -iE 'conditions|events|readiness'

Check the GKE operation or drain that’s blocked, and the autoscaler/upgrade events:

gcloud container operations list --filter="TYPE:UPGRADE_NODES AND STATUS:RUNNING"
kubectl get events -A --field-selector reason=EvictionBlocked --sort-by=.lastTimestamp

Example Root Cause Analysis

A scheduled GKE node pool upgrade stalled for hours. gcloud container operations list showed the UPGRADE_NODES operation still RUNNING, and Cloud Logging repeated the disruption-budget message for payments-api in prod.

kubectl get pdb -n prod showed payments-api with minAvailable: 2 and ALLOWED DISRUPTIONS: 0. The deployment was set to 2 replicas, so at full health the PDB permits zero disruptions by design — but the real problem was worse: kubectl get pods showed one of the two pods in CrashLoopBackOff after a bad config change. With only one Ready pod against a minAvailable: 2 budget, the drain could never proceed, and it never would until the crash was fixed.

The immediate fix was to resolve the CrashLoop (roll back the config), which restored the second Ready pod. With two healthy replicas the team temporarily scaled the deployment to 3 so ALLOWED DISRUPTIONS became 1, let the upgrade drain nodes one at a time, then scaled back. The durable fix was to right-size the PDB relative to replica count — minAvailable: 2 on a 2-replica deployment guarantees drains block, so they moved to 3 replicas with a maxUnavailable: 1 PDB, which tolerates a single node draining while preserving availability.

Prevention Best Practices

• Never set minAvailable equal to replica count. A 2-replica deployment with minAvailable: 2 (or maxUnavailable: 0) blocks every voluntary disruption, including upgrades. Leave headroom.
• Prefer maxUnavailable (e.g. maxUnavailable: 1) over minAvailable for stateless workloads; it scales naturally with replica count.
• Run at least 2–3 replicas for anything protected by a PDB so a single eviction is always permissible.
• Keep pods healthy. Alert on readyReplicas < replicas; an unready pod silently drives ALLOWED DISRUPTIONS to 0 and freezes upgrades.
• Audit for orphaned PDBs whose selectors match no current, healthy workload.
• Set surge upgrade settings (--max-surge, --max-unavailable) on node pools so upgrades add capacity before draining, giving PDBs room to satisfy.
• Configure maintenance windows and test drains in staging so a PDB misconfiguration surfaces before a production upgrade.

Quick Command Reference

# Which PDB blocks, and how many disruptions it allows
kubectl get pdb -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ALLOWED:.status.disruptionsAllowed

# PDB detail
kubectl describe pdb payments-api -n prod

# Actual healthy replicas vs desired
kubectl get deploy payments-api -n prod \
  -o custom-columns=NAME:.metadata.name,DESIRED:.spec.replicas,READY:.status.readyReplicas

# Temporarily add headroom so a drain can proceed
kubectl scale deploy payments-api -n prod --replicas=3

# Watch a blocked node pool upgrade
gcloud container operations list --filter="TYPE:UPGRADE_NODES AND STATUS:RUNNING"

# Retry the drain once healthy
kubectl drain gke-node-xyz --ignore-daemonsets --delete-emptydir-data

Conclusion

This error is the PodDisruptionBudget doing exactly its job: refusing to take a workload below its declared minimum availability. The trap is that a drain — and therefore a whole node pool upgrade — will hang forever when the budget can mathematically never be satisfied, most often because minAvailable equals the replica count or because an unready pod has quietly driven allowed disruptions to zero. Fix the blocked eviction by restoring healthy replicas or adding temporary headroom, then prevent recurrence by right-sizing PDBs against replica count and using surge upgrades so GKE always has room to drain one node at a time.