🚨 GCP Error: 'Your cluster's nodes are unhealthy' — Cause, Fix, and Troubleshooting Guide

Overview

GKE reports this when nodes in a pool fail to register or repeatedly fail their health checks, so the pool cannot reach a healthy state:

Warning: Your cluster's nodes are unhealthy.
Node pool "default-pool" has 2 nodes that are not ready.

$ kubectl get nodes
NAME                              STATUS     ROLES    AGE   VERSION
gke-prod-default-pool-a1b2-c3d4   NotReady   <none>   6m    v1.29.5-gke.100
gke-prod-default-pool-a1b2-e5f6   NotReady   <none>   6m    v1.29.5-gke.100

A NotReady node means the kubelet is not reporting healthy to the control plane. GKE node auto-repair may keep recreating the node, producing a churning pool that never stabilizes.

Symptoms

• kubectl get nodes shows NotReady, or nodes disappear and reappear.
• Console/gcloud shows the node pool in a RECONCILING or ERROR state.
• Pods stay Pending because there are no schedulable nodes.
• Node auto-repair events repeatedly recreate instances.

Common Root Causes

1. Nodes cannot reach the control plane / metadata server

Overly strict VPC firewall rules, a broken route, or Private Google Access disabled prevents the kubelet from registering.

2. Custom node config or DaemonSet breaks the kubelet

A bad DaemonSet, sysctl, or startup script crashes networking/CNI on every node.

3. Disk pressure or resource exhaustion

Node boot/ephemeral disk fills, tripping DiskPressure and eviction, which flips nodes NotReady.

4. Wrong / missing service account or scopes on the node pool

The node service account lacks roles/container.nodeServiceAccount or the pool has insufficient scopes, so nodes can’t pull config/images.

How to Diagnose

All read-only.

# Cluster and node-pool status
gcloud container clusters describe prod \
  --region=us-central1 --project=acme-prod-platform \
  --format="value(status,nodePools[].status)"

# Node conditions — what exactly is unhealthy?
kubectl describe node gke-prod-default-pool-a1b2-c3d4 | sed -n '/Conditions:/,/Addresses:/p'

# Recent node/kubelet events
kubectl get events -A --sort-by=.lastTimestamp | tail -30

# Node service account + scopes
gcloud container node-pools describe default-pool \
  --cluster=prod --region=us-central1 --project=acme-prod-platform \
  --format="value(config.serviceAccount, config.oauthScopes)"

# Firewall rules that could block node<->control-plane / metadata
gcloud compute firewall-rules list \
  --filter="network=prod-vpc" --project=acme-prod-platform \
  --format="table(name, direction, allowed[].map().firewall_rule().list(), disabled)"

The node Conditions block (e.g. DiskPressure=True, NetworkUnavailable=True, Ready=False) points straight at the category of problem.

Fixes

• Restore connectivity. Ensure firewall rules allow node-to-control-plane and egress to metadata.google.internal; enable Private Google Access on the node subnet if nodes have no external IP.
• Roll back the offending DaemonSet / node config you deployed just before nodes went NotReady:

kubectl rollout undo daemonset/my-cni-tweak -n kube-system

• Relieve disk pressure. Increase the node boot disk size (recreate the pool with a larger --disk-size) or clean up image/log bloat; add ephemeral storage limits to noisy pods.
• Fix the node service account. Grant the standard role and use the cloud-platform scope:

gcloud projects add-iam-policy-binding acme-prod-platform \
  --member="serviceAccount:gke-node@acme-prod-platform.iam.gserviceaccount.com" \
  --role="roles/container.nodeServiceAccount"

• Let auto-repair finish once the underlying cause is fixed, or manually recreate the pool.

What to Watch Out For

• Node auto-repair can mask a persistent root cause by endlessly recreating nodes — check why the health check fails, don’t just keep repairing.
• A recently applied DaemonSet, sysctl, or Config Sync change is the most common “everything went NotReady at once” trigger.
• Using the default Compute Engine service account with broad scopes is discouraged; a dedicated least-privilege node SA is safer but must have roles/container.nodeServiceAccount.
• Private clusters need Private Google Access + correct firewalls or nodes silently fail to register.

Related

• GCP Error: ‘GKE node evicted: DiskPressure’
• GCP Error: ‘0/n nodes are available: Insufficient cpu’
• GCP Error: ‘You must be logged in to the server (Unauthorized)’
• More in the GCP error guides.