☸️ Kubernetes Error Guide: 'failed to allocate for range 0: no IP addresses available' CNI IPAM Exhaustion

Overview

Warning  FailedCreatePodSandBox  18s (x8 over 3m)  kubelet
  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox
  "a1b2...": plugin type="host-local" failed (add): failed to allocate for range 0:
  no IP addresses available in range set: 10.244.3.1-10.244.3.254

This is a CNI IPAM (IP Address Management) exhaustion error. Every pod needs an IP, and the CNI plugin’s IPAM component hands them out from a range assigned to the node — typically a per-node slice of the cluster pod CIDR (with host-local) or an ENI/subnet (with the AWS VPC CNI). When that range is fully allocated, the plugin cannot give a new pod an address, sandbox creation fails, and the pod is stuck in ContainerCreating.

The range is genuinely full or it only looks full because IPs have leaked — freed pods whose IPAM entries were never reclaimed. Either way the symptom is identical: new pods on the affected node cannot start, while existing pods keep running fine.

Symptoms

• Pods stay ContainerCreating on specific nodes, never reaching Running.
• kubectl describe pod shows FailedCreatePodSandBox with no IP addresses available in range.
• The failure clusters on the fullest nodes; emptier nodes schedule fine.
• kubectl get pods -o wide shows the node already near its pod capacity.
• After heavy pod churn (CI jobs, CronJobs), the range fills faster than it drains.

Common Root Causes

1. The per-node pod CIDR is too small

A /24 node CIDR gives ~254 IPs. Nodes running many pods, or with high maxPods, exhaust that quickly.

2. Leaked IP allocations

Pods were deleted but their host-local IPAM entries in /var/lib/cni/networks/ were never removed (often after a kubelet or containerd crash), so the range shows full while few pods exist.

3. Cloud subnet exhaustion (AWS VPC CNI)

With the VPC CNI, pod IPs come from the VPC subnet. A small subnet, or many nodes/ENIs, drains available secondary IPs and IPAM fails cluster-wide.

4. Undersized cluster pod CIDR

The overall cluster CIDR (e.g. 10.244.0.0/16 split into /24 per node) caps how many nodes and pods can ever get addresses.

5. Rapid pod churn outpacing IP release

Short-lived Jobs/CronJobs create and destroy pods faster than IPAM reclaims addresses, especially if teardown is not graceful.

Diagnostic Workflow

Step 1: Confirm the failing pods and node

kubectl get pods -A -o wide | grep ContainerCreating
kubectl describe pod <pod> -n <namespace> | grep -i "no IP addresses\|FailedCreatePodSandBox"

Step 2: Check the node’s pod CIDR and capacity

kubectl get node <node> -o jsonpath='{.spec.podCIDR}{"\n"}'
kubectl get node <node> -o jsonpath='{.status.capacity.pods}{"\n"}'
kubectl get pods -A -o wide --field-selector spec.nodeName=<node> | wc -l

Step 3: Inspect host-local IPAM allocations on the node

sudo ls /var/lib/cni/networks/<network-name>/ | wc -l
sudo crictl pods | wc -l

If the allocation count far exceeds running sandboxes, IPs have leaked.

Step 4: For AWS VPC CNI, check subnet and ENI headroom

kubectl logs -n kube-system -l k8s-app=aws-node --tail=50 | grep -i "insufficient\|no available\|ipamd"

Step 5: Review the CNI config for range size

sudo cat /etc/cni/net.d/*.conflist | grep -A3 "subnet\|rangeStart\|rangeEnd"

Step-by-Step Resolution

1. Reclaim leaked IPs first — the fastest win. Compare IPAM entries to live sandboxes; if entries are stale, restart the runtime and CNI so IPAM re-syncs:

   sudo crictl pods
   sudo ls /var/lib/cni/networks/<network-name>/

   After confirming a pod is truly gone, remove its stale reservation file, then:

   sudo systemctl restart containerd
   kubectl -n kube-system rollout restart daemonset <cni-daemonset>

2. Spread the load by scaling out nodes so pods schedule onto ranges with free addresses:

   kubectl get nodes
   # scale your node group / cluster-autoscaler up

3. Enlarge the per-node CIDR if /24 is too small. This is a cluster-level change: increase the node CIDR mask (for example --node-cidr-mask-size=23) so each node gets ~510 IPs, and ensure the cluster pod CIDR is big enough.

4. Add subnet capacity for the AWS VPC CNI — attach a larger secondary CIDR or additional subnets to the VPC, or enable prefix delegation so each ENI serves a /28 block of pod IPs.

5. Cap pods per node with maxPods so scheduling never promises more IPs than the range holds:

   sudo grep maxPods /var/lib/kubelet/config.yaml

6. Verify recovery — pending pods should schedule:

   kubectl get pods -n <namespace> -w

Prevention

• Size the cluster pod CIDR and per-node CIDR mask for peak pod count plus headroom before the cluster grows into the limit.
• Monitor IP utilization per node/subnet and alert well before ranges saturate.
• For the AWS VPC CNI, enable prefix delegation and keep spare subnet capacity for scale-outs.
• Ensure graceful pod termination so IPAM reliably releases addresses; watch for host-local leak after runtime crashes.
• Set maxPods consistent with the addressable range so the scheduler never over-commits IPs.

Related Errors

• Pod sandbox changed, it will be killed and re-created — the repeated sandbox failures IPAM exhaustion causes.
• failed to setup network for sandbox — the parent CNI setup failure that wraps this IPAM message.
• InsufficientFreeAddressesInSubnet (AWS) — the VPC-level flavor of the same exhaustion.
• 0/N nodes are available: node(s) had untolerated taint — a scheduling failure that looks similar but is unrelated to IPs.

Frequently Asked Questions

Is the range really full or are IPs leaked? Compare ls /var/lib/cni/networks/<network>/ | wc -l against crictl pods | wc -l on the node. If reservations vastly exceed running sandboxes, you have leaked IPs, and restarting the runtime plus CNI usually reclaims them.

Can I fix this without recreating the cluster? Often yes — reclaim leaked IPs, scale out nodes, or add subnet capacity (AWS). Enlarging the per-node CIDR mask can be done on the controller manager, though the cluster pod CIDR itself is fixed at creation.

Why does it only hit some nodes? IPAM is per-node with host-local. The busiest nodes exhaust their slice first while emptier nodes still have free addresses, so failures cluster on the fullest ones.

How do I stop CronJob churn from exhausting IPs? Ensure jobs terminate cleanly so addresses release, cap concurrency, and give the range enough headroom for peak simultaneous pods. Generate a capacity plan with the DevOps AI prompt library.

Where can I learn more? See the Kubernetes & Helm guides.