🚨 AWS Error: EKS worker nodes stuck in 'NotReady' — Cause, Fix, and Troubleshooting Guide

Overview

A Kubernetes node reports Ready only after its kubelet registers with the control plane, the CNI (on EKS, the aws-node VPC CNI) initializes pod networking, and no node-level pressure conditions are set. When any of those fail, the node registers but stays NotReady, so the scheduler won’t place pods on it and workloads stay Pending.

You will see it from kubectl:

NAME                            STATUS     ROLES    AGE   VERSION
ip-10-0-3-51.ec2.internal       NotReady   <none>   6m    v1.29.6-eks-REDACTED

kubectl describe node shows the underlying condition, e.g.:

Ready   False   KubeletNotReady   container runtime network not ready: NetworkReady=false ... cni plugin not initialized

It occurs when the VPC CNI can’t run, the node’s IAM role lacks required policies, the node can’t reach the control-plane endpoint, or the subnet has no free IPs.

Symptoms

• kubectl get nodes shows one or more nodes NotReady; pods stay Pending.
• aws-node and/or kube-proxy pods are CrashLoopBackOff in kube-system.
• The managed node group shows DEGRADED or nodes flap between Ready/NotReady.
• New nodes from an autoscaler never become schedulable.

Common Root Causes

1. The VPC CNI (aws-node) can’t initialize

aws-node is crashing or can’t attach ENIs/IPs, so the node’s network never becomes ready.

2. The node IAM role is missing required policies

Nodes need AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly, and AmazonEKS_CNI_Policy. Missing the CNI policy stops IP assignment.

3. Security groups / NACLs block the control plane

The node can’t reach the EKS API endpoint (443) or the control plane can’t reach the kubelet (10250), so registration/health checks fail.

4. Subnet IP exhaustion

The CNI can’t allocate secondary IPs/ENIs because the subnet is out of addresses.

5. Node pressure conditions

DiskPressure, MemoryPressure, or PIDPressure (e.g. a full disk on /var/lib/containerd) flip the node NotReady.

How to diagnose

Step 1: Read the node’s conditions and events

kubectl describe node ip-10-0-3-51.ec2.internal | sed -n '/Conditions:/,/Events:/p'

The Ready condition’s Reason/Message (CNI not initialized, disk pressure, kubelet stopped posting status) points at the layer to fix.

Step 2: Check the system daemonsets

kubectl get pods -n kube-system -o wide | grep -E 'aws-node|kube-proxy'
kubectl logs -n kube-system ds/aws-node -c aws-node --tail=50

CNI errors here (e.g. failed to assign an IP address) confirm networking or IAM/subnet causes.

Step 3: Verify the node role and networking

aws eks describe-nodegroup --cluster-name prod --nodegroup-name ng-1 \
  --query 'nodegroup.[nodeRole,health.issues]' --output json

ROLE=$(aws eks describe-nodegroup --cluster-name prod --nodegroup-name ng-1 \
  --query 'nodegroup.nodeRole' --output text | awk -F/ '{print $NF}')
aws iam list-attached-role-policies --role-name "$ROLE" --output table

Step 4: Check free IPs in the node subnets

aws ec2 describe-subnets --subnet-ids subnet-REDACTED \
  --query 'Subnets[].[SubnetId,AvailableIpAddressCount]' --output table

Fixes

Attach the missing node IAM policies

for p in AmazonEKSWorkerNodePolicy AmazonEC2ContainerRegistryReadOnly AmazonEKS_CNI_Policy; do
  aws iam attach-role-policy --role-name "$ROLE" --policy-arn arn:aws:iam::aws:policy/$p
done

Restore CNI networking

Ensure aws-node is healthy (update the VPC CNI add-on if outdated) and recycle affected nodes:

aws eks update-addon --cluster-name prod --addon-name vpc-cni --resolve-conflicts OVERWRITE
kubectl rollout restart ds/aws-node -n kube-system

Open control-plane connectivity and free up IPs

Allow node↔control-plane traffic (443 outbound to the cluster SG, 10250 inbound from it), and relieve exhaustion by adding a larger/second subnet or enabling prefix delegation on the CNI.

Clear disk pressure

Increase the node volume size in the launch template, or clean image/log buildup, then replace the node.

What to watch out for

• The Ready condition message names the failing layer — always read it before guessing (CNI vs. disk vs. kubelet).
• A missing AmazonEKS_CNI_Policy looks like a networking bug but is really IAM — check attached policies early.
• IP exhaustion scales with pod density; prefix delegation or larger subnets prevents recurrence.
• Nodes that pass registration but never go Ready are almost always CNI/IAM/SG, not the Kubernetes version.

Related

• AWS Error: EKS ‘You must be logged in to the server (Unauthorized)’ — the control-plane auth counterpart.
• AWS Error: ‘not enough free addresses in subnet’ (ENI) — subnet IP exhaustion that also starves the CNI.
• AWS Error: ‘is not authorized to perform: sts:AssumeRole’ — IRSA/role issues affecting node and pod identities.