🚨 Azure Error: AKS node 'NotReady' — Cause, Fix, and Troubleshooting Guide

Overview

An AKS node reports NotReady when the Kubernetes control plane has stopped receiving healthy heartbeats from that node’s kubelet. Pods on the node are no longer scheduled or are evicted, and after the eviction timeout the scheduler tries to move workloads elsewhere. The literal state shows up in kubectl get nodes:

NAME                                STATUS     ROLES   AGE   VERSION
aks-nodepool1-12345678-vmss000000   Ready      agent   40d   v1.29.4
aks-nodepool1-12345678-vmss000001   NotReady   agent   40d   v1.29.4

And the node’s conditions carry the reason, e.g. KubeletNotReady, DiskPressure, or a stale Ready heartbeat:

Conditions:
  Type             Status    Reason
  Ready            Unknown   NodeStatusUnknown  (Kubelet stopped posting node status.)
  MemoryPressure   Unknown   NodeStatusUnknown
  DiskPressure     Unknown   NodeStatusUnknown

NodeStatusUnknown means the control plane simply stopped hearing from the node; the reason on the individual conditions tells you whether it is a resource, network, or kubelet problem.

Symptoms

• One or more nodes show NotReady (or NotReady,SchedulingDisabled) in kubectl get nodes.
• Pods on the affected node move to Terminating, Unknown, or get rescheduled; new pods land only on healthy nodes.
• az aks show reports the cluster or node pool in a Failed or degraded provisioning state.
• Intermittent flapping between Ready and NotReady under load.

Common Root Causes

• Resource pressure — the node hit DiskPressure (full OS/container disk) or MemoryPressure, and kubelet stopped reporting healthy.
• Kubelet or containerd crash — the node agent died or is wedged, so no heartbeat reaches the API server.
• Network path to the API server broken — an NSG/UDR/firewall change blocks the node’s egress to the AKS control plane or required endpoints.
• CNI/IP exhaustion — with Azure CNI, the subnet ran out of IPs so the node cannot configure pod networking.
• Node image/kernel issue after an upgrade — a bad node image or a stuck OS update.
• Underlying VMSS instance unhealthy — the backing virtual machine scale set instance is stopped, deallocated, or failed.

How to diagnose

Get the node and read its conditions and events:

kubectl get nodes -o wide
kubectl describe node aks-nodepool1-12345678-vmss000001

The Conditions and Events sections at the bottom of describe usually name the reason (DiskPressure, KubeletNotReady, network timeouts).

Check the node pool and cluster provisioning state:

az aks nodepool show \
  --resource-group aks-rg --cluster-name prod-aks --name nodepool1 \
  --query "{state:provisioningState, count:count, vmSize:vmSize}" -o json

If Azure CNI, confirm the pod subnet is not out of IPs:

az network vnet subnet show \
  --resource-group net-rg --vnet-name aks-vnet --name pod-subnet \
  --query "{prefix:addressPrefix, ipConfigs:length(ipConfigurations)}" -o json

Run the built-in AKS diagnostics, which check node health, DNS, and outbound connectivity:

az aks kollect --resource-group aks-rg --name prod-aks   # collects node logs (Periscope)

Confirm required outbound endpoints are reachable (control plane, MCR, etc.) by checking your NSG/UDR and firewall rules against the AKS egress requirements.

Fixes

Resource pressure — free disk/memory or scale the node pool. Cordon and drain, then delete the wedged node so the VMSS replaces it:

kubectl cordon aks-nodepool1-12345678-vmss000001
kubectl drain aks-nodepool1-12345678-vmss000001 \
  --ignore-daemonsets --delete-emptydir-data

Let AKS reimage/replace the node. For a persistently unhealthy node, reconcile the node pool, which reimages failed instances:

az aks nodepool upgrade \
  --resource-group aks-rg --cluster-name prod-aks --name nodepool1 \
  --node-image-only

IP exhaustion (Azure CNI) — add address space or scale down pod density; a larger pod subnet or maxPods tuning prevents recurrence.

Broken egress — restore the outbound path (NSG allow rules, UDR next-hop, firewall FQDN allowlist) so kubelet can reach the control plane again.

Recover a stuck cluster provisioning state with a no-op reconcile:

az aks update --resource-group aks-rg --name prod-aks

After the fix, confirm the node returns to Ready:

kubectl get nodes -w

What to watch out for

• Auto-repair takes time. AKS node auto-repair will reboot/reimage an unhealthy node, but only after a detection window; do not assume an instant fix.
• Draining with PodDisruptionBudgets can hang if a PDB is too strict — check kubectl get pdb before draining production.
• NotReady after an upgrade points at the node image or a workload that mis-behaves on the new kubelet; roll the node pool image rather than the whole cluster if isolated.
• Egress requirements change. Locking down outbound traffic without allowing AKS’s required FQDNs is a top cause of nodes going NotReady days after a “successful” firewall change.

Related

• AKS ImagePullBackOff — pod-level failure that often accompanies node or registry issues.
• Troubleshooting AKS with AI — broader AKS diagnostic workflow and prompts.
• AllocationFailed — when the backing VMSS cannot get capacity to replace an unhealthy node.