🚨 GCP Error: 'node(s) had untolerated taint' — Cause, Fix, and Troubleshooting Guide

Overview

GKE reports this when a pod cannot be scheduled because every candidate node carries a taint the pod does not tolerate, so the scheduler filters those nodes out and the pod stays Pending:

$ kubectl describe pod batch-9f2c
...
Events:
  Type     Reason            Message
  ----     ------            -------
  Warning  FailedScheduling  0/4 nodes are available:
                             2 node(s) had untolerated taint {dedicated: gpu},
                             2 node(s) had untolerated taint
                             {node.kubernetes.io/not-ready: }.

Taints repel pods; tolerations let specific pods land on tainted nodes. A mismatch (or a transient system taint like not-ready) leaves the pod unschedulable.

Symptoms

• kubectl describe pod shows node(s) had untolerated taint {...}.
• Pods pin to a dedicated pool (GPU/Spot) but lack the matching toleration.
• Pods won’t schedule during a node-not-ready window or an ongoing upgrade.
• New workloads never land on tainted node pools you created for isolation.

Common Root Causes

1. Dedicated node pool taint without a matching toleration

A pool created with --node-taints=dedicated=gpu:NoSchedule repels every pod that doesn’t tolerate dedicated=gpu.

2. Spot / preemptible taint

Spot pools carry cloud.google.com/gke-spot=true:NoSchedule; workloads must tolerate it explicitly.

3. Transient system taints

node.kubernetes.io/not-ready or unreachable during node startup, upgrades, or health blips.

4. Missing nodeSelector/affinity for a tainted-and-labeled pool

Toleration alone lets a pod land there but doesn’t attract it; without a selector it may still not target the intended pool.

How to Diagnose

All read-only.

# Exact FailedScheduling reason
kubectl describe pod batch-9f2c | sed -n '/Events:/,$p'

# What taints do the nodes carry?
kubectl get nodes -o custom-columns=\
'NAME:.metadata.name,TAINTS:.spec.taints[*].key' 

# Node-pool taint definition on GKE
gcloud container node-pools describe gpu-pool \
  --cluster=prod --region=us-central1 --project=acme-prod-platform \
  --format="value(config.taints[].key, config.taints[].value, config.taints[].effect)"

# The pod's current tolerations
kubectl get pod batch-9f2c -o jsonpath='{.spec.tolerations}' | tr ',' '\n'

Match the node taint key=value:effect against the pod’s tolerations — the gap is your fix.

Fixes

Add a matching toleration (and usually a nodeSelector to target the pool):

spec:
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "gpu"
      effect: "NoSchedule"
  nodeSelector:
    cloud.google.com/gke-nodepool: "gpu-pool"

For Spot pools, tolerate the spot taint:

  tolerations:
    - key: "cloud.google.com/gke-spot"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

Transient not-ready/unreachable taints usually clear on their own once the node registers or the upgrade finishes — check node health if they persist (see the nodes-unhealthy guide).

Remove an unintended taint you added by mistake:

kubectl taint nodes gke-prod-gpu-a1b2 dedicated=gpu:NoSchedule-

What to Watch Out For

• A toleration permits scheduling onto a tainted node but does not require it — pair it with a nodeSelector/affinity to actually land on the intended pool.
• Spot and GPU pools carry taints by design; every workload targeting them must tolerate those taints.
• Persistent node.kubernetes.io/not-ready taints point to unhealthy nodes, not a toleration bug — investigate node health.
• This reason is distinct from Insufficient cpu; read the full message, which may list several reasons per node group.

Related

• GCP Error: ‘0/n nodes are available: Insufficient cpu’
• GCP Error: ‘Your cluster’s nodes are unhealthy’
• GCP Error: ‘GKE node evicted: DiskPressure’
• More in the GCP error guides.