🚨 Kubernetes Error Guide: 'CNI request failed with status 400' failed to delegate add

Exact Error Message

A pod is stuck in ContainerCreating and the kubelet records a FailedCreatePodSandBox event whose CNI delegate call returned HTTP 400:

Events:
  Type     Reason                  Age   From     Message
  ----     ------                  ----  ----     -------
  Warning  FailedCreatePodSandBox  6s    kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a1b2...": plugin type="calico" failed (add): error adding host side routes for interface: cali3f...: networkPlugin cni failed to set up pod "web-7d9f_default" network: CNI request failed with status 400: 'failed to delegate add: failed to allocate for range 0: no IP addresses available in range set'

The pod cannot get a network namespace, so it never starts. The decisive phrase is usually inside failed to delegate add — here, no IP addresses available.

What the Error Means

When the kubelet creates a pod sandbox, it invokes the CNI plugin named in /etc/cni/net.d. Many CNIs (Calico, Cilium, Multus, Flannel via a meta-plugin) act as a thin front end that delegates the actual IP allocation to an IPAM plugin or a local agent over a small HTTP/gRPC API. CNI request failed with status 400 is that delegated call returning a client-error status, and failed to delegate add confirms the front-end-to-IPAM hand-off is where it broke.

Status 400 means the request reached the agent but was rejected as unfulfillable — most commonly because the IPAM pool is exhausted, the config on disk is invalid, or the agent’s datastore is unreachable. It is distinct from a missing config (no plugins at all), which yields a different “cni plugin not initialized” error.

Common Causes

• IPAM pool exhaustion — no IP addresses available in range set; the node’s pod CIDR block or IP pool is fully allocated, often due to leaked IPs from ungracefully deleted pods.
• Stale CNI config or multiple configs — leftover files in /etc/cni/net.d make the kubelet pick the wrong or a broken plugin.
• CNI agent not ready — the Calico/Cilium agent DaemonSet pod on that node is crash-looping or still initializing.
• Datastore unreachable — Calico’s etcd/Kubernetes datastore or Cilium’s KV store is unreachable, so allocation requests fail.
• Node CIDR not assigned — the node never received a podCIDR (controller-manager --allocate-node-cidrs misconfig), so the range set is empty.
• MTU / config mismatch after upgrade — a partially upgraded CNI leaves an incompatible binary and config combination.

How to Reproduce the Error

Shrink a Flannel/host-local pod CIDR to a tiny range, then schedule more pods than it holds:

# A /30 leaves room for only a couple of pod IPs per node
# (set in the CNI's IPAM "subnet"/"ipam.ranges" config), then:
kubectl create deployment ipfill --image=registry.k8s.io/pause:3.9 --replicas=20
kubectl get pods -l app=ipfill -o wide

Once the range is full, new pods stick in ContainerCreating and kubectl describe shows CNI request failed with status 400: ... no IP addresses available in range set.

Diagnostic Commands

# Read the full sandbox/CNI failure on the pod
kubectl describe pod <POD> | grep -A8 Events

# Is the CNI agent healthy on the pod's node?
kubectl get pods -n kube-system -o wide | grep -E 'calico|cilium|flannel' | grep <NODE>

# Inspect the CNI config the kubelet will use (read-only, on the node)
ls -l /etc/cni/net.d/
journalctl -u kubelet --since "10 min ago" | grep -i cni

# Confirm the node actually has a podCIDR assigned
kubectl get node <NODE> -o jsonpath='{.spec.podCIDR}{"\n"}'

# Container runtime view of the failed sandbox
crictl pods --name <POD> 2>/dev/null

The combination of the describe message and the CNI agent pod’s logs on that specific node usually pinpoints the cause.

Step-by-Step Resolution

1. Read the delegated error. The text after failed to delegate add: is the real cause. no IP addresses available is exhaustion; invalid or failed to load points at config; a datastore/dial error points at the agent’s backend.

2. Fix IPAM exhaustion. If IPs are exhausted, first reclaim leaked allocations. Ungracefully deleted pods can leave dangling IPs; the CNI agent usually has a tool to list/reclaim them (e.g. Calico’s IPAM commands). Then enlarge the pool — increase the cluster pod CIDR / IP pool block size or add nodes so allocations spread out.

3. Clean stale CNI config. Multiple files in /etc/cni/net.d cause the kubelet to load the lexicographically first one, which may be obsolete. Ensure only the intended plugin’s config remains, then the kubelet picks up the change automatically.

4. Restart / heal the CNI agent. If the agent DaemonSet pod on that node is CrashLoopBackOff or not Ready, fix or restart it; no agent means no delegate target. Check its logs for datastore connection errors.

5. Restore the datastore connection. For Calico/Cilium, confirm the agent can reach its datastore (Kubernetes API, etcd, or KV store). Network policy, expired certs, or a down etcd will make every allocate request fail.

6. Ensure node CIDRs are assigned. If spec.podCIDR is empty, the controller-manager is not allocating CIDRs. Confirm --allocate-node-cidrs=true and --cluster-cidr are set, then the node will receive a range.

7. Verify recovery. After the fix, delete the stuck pod so it reschedules with a fresh sandbox:

kubectl delete pod <POD>
kubectl get pod -l <SELECTOR> -o wide -w

Prevention and Best Practices

• Size the cluster pod CIDR and per-node block size for peak pod density plus headroom; exhaustion is the top cause of this error.
• Monitor free IPs per pool/node and alert before the pool fills.
• Keep exactly one CNI config in /etc/cni/net.d and manage it through the CNI’s installer, not by hand.
• Upgrade CNI binary and config together; never leave a half-upgraded plugin on a node.
• Watch CNI agent DaemonSet health and datastore connectivity as first-class cluster signals. More in the Kubernetes & Helm guides.

Related Errors

• networkPlugin cni failed to set up pod — the broader CNI setup failure family.
• FailedCreatePodSandBox — the umbrella sandbox-creation event.
• dial tcp i/o timeout — connectivity failures after the pod gets an IP.
• lookup no such host on cluster.local — DNS failures once networking is up.

Frequently Asked Questions

Does status 400 mean my YAML is wrong? No. The 400 is between the CNI front-end and its IPAM backend, not about your pod manifest. Read the failed to delegate add text — it is almost always IP exhaustion, a config issue, or an unreachable agent datastore.

Why only some nodes? IPAM is often per-node (each node gets a block of the pod CIDR). One node can exhaust its block while others have room, and a crash-looping agent affects only the node it runs on. That is why the error frequently clusters on specific nodes.

I deleted the stuck pods but IPs are still exhausted. Why? Ungraceful deletions can leak IP allocations that the agent has not reclaimed. Use the CNI’s IPAM tooling to list and release orphaned handles, then the freed IPs become available again.

Will restarting the CNI agent lose running pods’ networking? Restarting the agent DaemonSet pod generally does not tear down existing pod networking — datapath state persists — but follow your CNI’s documented restart guidance to be safe.

How is this different from cni plugin not initialized? That error means the kubelet found no usable CNI config at all (empty/invalid /etc/cni/net.d). CNI request failed with status 400 means the plugin loaded and ran but its delegated allocation was rejected — a later, more specific stage of the same pipeline.