🚨 GCP Error: '0/n nodes are available: Insufficient cpu' — Cause, Fix, and Troubleshooting Guide

Overview

The GKE scheduler emits this when no node has enough allocatable CPU (or memory) left to satisfy a pod’s resource requests, leaving the pod Pending:

$ kubectl describe pod api-7c9f-xzkq
...
Events:
  Type     Reason            Message
  ----     ------            -------
  Warning  FailedScheduling  0/3 nodes are available: 3 Insufficient cpu.
                             preemption: 0/3 nodes are available:
                             3 No preemption victims found for incoming pod.

This is about requests vs. allocatable, not actual usage. A node can look 40% utilized yet still be “full” because reservations already sum to its allocatable CPU.

Symptoms

• Pods stuck Pending; kubectl describe pod shows Insufficient cpu (or Insufficient memory).
• Deployments never reach the desired replica count.
• New pods schedule fine on some nodes but the cluster as a whole is out of headroom.
• Cluster Autoscaler logs no.scale.up or scale-up disabled.

Common Root Causes

1. Pod CPU requests too high for any node

A single pod requests more CPU than a node’s allocatable amount, so it can never fit anywhere.

2. Cumulative requests fill the pool

Total requests across running pods have consumed all allocatable CPU; no node has room for one more.

3. Cluster Autoscaler can’t add nodes

Autoscaling is off, at max size, or blocked by quota/ZONE_RESOURCE_POOL_EXHAUSTED.

4. Node allocatable smaller than expected

System/kube-reserved and eviction thresholds reduce allocatable below the machine’s nominal vCPUs, so “8 vCPU” nodes offer less.

How to Diagnose

All read-only.

# Confirm the reason and the pod's requests
kubectl describe pod api-7c9f-xzkq | sed -n '/Requests:/,/Events:/p'

# Allocatable vs requested per node
kubectl describe nodes | grep -A6 "Allocated resources"

# Node capacity/allocatable
kubectl get nodes -o custom-columns=\
'NAME:.metadata.name,CPU_ALLOC:.status.allocatable.cpu,CPU_CAP:.status.capacity.cpu'

# Is the autoscaler enabled and what are the bounds?
gcloud container node-pools describe default-pool \
  --cluster=prod --region=us-central1 --project=acme-prod-platform \
  --format="value(autoscaling.enabled, autoscaling.minNodeCount, autoscaling.maxNodeCount)"

Compare the pod’s cpu request to per-node allocatable. If the request exceeds allocatable, no scaling will help until you shrink the request or use a bigger machine type.

Fixes

• Right-size the request so the pod fits the nodes:

resources:
  requests:
    cpu: "500m"     # was 4 — set to what the app actually needs
    memory: "512Mi"

• Scale the node pool (or enable/raise autoscaling):

gcloud container clusters resize prod \
  --node-pool=default-pool --num-nodes=5 \
  --region=us-central1 --project=acme-prod-platform

gcloud container clusters update prod \
  --enable-autoscaling --min-nodes=2 --max-nodes=10 \
  --node-pool=default-pool --region=us-central1 --project=acme-prod-platform

• Add a larger machine type node pool for genuinely big pods.
• Clear autoscaler blockers: raise CPU quota, or fail over to a zone with capacity if scale-up hits ZONE_RESOURCE_POOL_EXHAUSTED.

What to Watch Out For

• The scheduler counts requests, not live usage — a node at low CPU utilization can still be “full.”
• Allocatable is always less than nominal vCPUs (system/kube-reserved + eviction threshold); size requests against allocatable.
• A single oversized request that exceeds any node’s allocatable is unschedulable forever — the fix is a smaller request or bigger node, never more nodes.
• Untolerated taints produce a different FailedScheduling reason; read the message carefully.

Related

• GCP Error: ‘node(s) had untolerated taint’ (Pod unschedulable)
• GCP Error: ‘ZONE_RESOURCE_POOL_EXHAUSTED’
• GCP Error: ‘IP_SPACE_EXHAUSTED’ on GKE
• More in the GCP error guides.