🚨 Kubernetes Error Guide: 'networkPlugin cni failed to set up pod' CNI Failures

Exact Error Message

A pod stays in ContainerCreating and its events show the CNI setup failing:

Warning  FailedCreatePodSandbox  18s  kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9f3c...": plugin type="calico" failed (add): networkPlugin cni failed to set up pod "web-6d8c9f7b5c-q4n2t_prod" network: error getting ClusterInformation: connection is unauthorized

Common variants of the same networkPlugin cni failed to set up pod message:

failed to find plugin "calico" in path [/opt/cni/bin]
failed to allocate for range 0: no IP addresses available in range set: 10.244.1.1-10.244.1.254
failed to set bridge addr: "cni0" already has an IP address different from 10.244.2.1/24

What the Error Means

When the kubelet creates a pod, it asks the container runtime to build a network sandbox by invoking the CNI plugin named in /etc/cni/net.d. The plugin binary (from /opt/cni/bin) sets up the pod’s network namespace, allocates an IP from its IPAM range, and wires it to the node bridge or overlay. If any of those steps fails, the runtime returns networkPlugin cni failed to set up pod ... network: <reason> and the pod is stuck in ContainerCreating with 0/1 ready. No container starts until the sandbox succeeds, and the kubelet retries the sandbox creation on a backoff.

The text after network: classifies the failure: a missing plugin binary, an exhausted IP pool, a stale or conflicting bridge, or an unreachable CNI control plane. Each maps to a distinct fix.

Common Causes

• CNI binaries or config missing. /etc/cni/net.d has no config, or /opt/cni/bin lacks the named plugin (failed to find plugin "..." in path).
• IPAM pool exhaustion. The node’s pod CIDR is fully allocated (no IP addresses available in range set), often from leaked IPs after ungraceful pod deletes.
• Stale cni0 / bridge after a CIDR change. The host bridge keeps an old address (cni0 already has an IP address different from ...) after the cluster pod CIDR changed.
• CNI plugin pods not Ready. The Calico/Cilium/Flannel DaemonSet pod on the node is crashing or not yet scheduled, so the datastore is unreachable (connection is unauthorized, connection refused).
• Node podCIDR not assigned. The controller-manager never assigned a spec.podCIDR to the node, so host-local IPAM has no range.
• MTU mismatch. An overlay MTU larger than the underlying network’s causes pod setup or later traffic to fail.

How to Reproduce the Error

Remove the CNI config on a node and create a pod scheduled there:

# On node-2 — hide the CNI config
sudo mv /etc/cni/net.d/10-calico.conflist /tmp/

# Schedule a pod onto node-2
kubectl run cni-test --image=nginx --overrides='{"spec":{"nodeName":"node-2"}}'
kubectl describe pod cni-test | grep -A4 Events

Warning  FailedCreatePodSandbox  3s  kubelet  Failed to create pod sandbox: ... networkPlugin cni failed to set up pod "cni-test_default" network: failed to find plugin "calico" in path [/opt/cni/bin]

Restoring the config file (sudo mv /tmp/10-calico.conflist /etc/cni/net.d/) lets the next sandbox attempt succeed.

Diagnostic Commands

# Pod status and the exact CNI message
kubectl get pod <POD> -o wide
kubectl describe pod <POD> | grep -A6 Events

# On the node — CNI config and binaries present?
ls -l /etc/cni/net.d/
ls -l /opt/cni/bin/

# Is the node's podCIDR assigned and the bridge sane?
kubectl get node <NODE> -o jsonpath='{.spec.podCIDR}{"\n"}'
ip addr show cni0 2>/dev/null
ip route | grep -i cni

# CNI plugin pods and runtime sandbox state
kubectl get pods -n kube-system -o wide | grep -Ei 'calico|cilium|flannel'
kubectl logs -n kube-system -l k8s-app=calico-node --tail=50
sudo crictl pods | grep NotReady
journalctl -u kubelet -n 60 --no-pager | grep -i cni

# IPAM exhaustion — count allocated IPs on the node
ls /var/lib/cni/networks/*/  2>/dev/null | wc -l

Step-by-Step Resolution

1. Classify by the text after network:. failed to find plugin is a binary/config problem; no IP addresses available is IPAM; cni0 already has an IP address is a stale bridge; connection is unauthorized/refused is an unready CNI control plane.

2. Restore missing CNI config and binaries. Reinstall the CNI DaemonSet (it ships the binaries into /opt/cni/bin) or restore the config file.

   ls /etc/cni/net.d/ /opt/cni/bin/
   kubectl rollout restart daemonset/calico-node -n kube-system

3. Fix an unready CNI pod. If the plugin pod is crashing, read its logs and restart it; many connection is unauthorized errors clear once the node agent re-establishes its datastore connection.

   kubectl get pods -n kube-system -o wide | grep calico-node
   kubectl logs -n kube-system <calico-node-pod> -c calico-node --tail=80
   kubectl delete pod -n kube-system <calico-node-pod>

4. Reclaim an exhausted IP pool. Delete stuck/terminating pods that leaked IPs, then clear stale host-local IPAM reservations and restart the kubelet.

   kubectl get pods -A --field-selector status.phase=Pending -o wide
   sudo ls /var/lib/cni/networks/<NETWORK>/   # stale IP files
   # remove leaked reservations only if the pods are truly gone, then:
   sudo systemctl restart kubelet

5. Repair a stale cni0 bridge after a CIDR change. Delete the bridge so the CNI recreates it with the correct address.

   sudo ip link set cni0 down
   sudo ip link delete cni0
   sudo systemctl restart kubelet

6. Assign a missing podCIDR. If spec.podCIDR is empty, confirm --allocate-node-cidrs=true and --cluster-cidr on kube-controller-manager, then recreate the node object.

   kubectl get node <NODE> -o jsonpath='{.spec.podCIDR}{"\n"}'

7. Verify recovery.

   kubectl delete pod <POD>   # force a fresh sandbox
   kubectl get pod <POD> -o wide -w

Prevention and Best Practices

• Install the CNI via its DaemonSet/operator so binaries land in /opt/cni/bin on every node automatically, including newly joined nodes.
• Size pod CIDRs generously; a /24 per node caps you at ~254 pods and exhausts quickly on dense nodes. Monitor IP allocation per node.
• Never change the cluster pod CIDR in place without recreating bridges — stale cni0 addresses cause bridge addr conflicts across the fleet.
• Alert on CNI DaemonSet readiness; an unready calico-node/cilium/flannel pod silently blocks every new pod on that node.
• Set the overlay MTU to match the underlying network (typically 1450 for VXLAN over 1500) and validate large-packet pod-to-pod traffic.
• Confirm --allocate-node-cidrs=true and a matching --cluster-cidr on the controller-manager so every node gets a podCIDR. See more in Kubernetes & Helm guides.

Related Errors

• FailedCreatePodSandbox — the parent event that wraps every CNI setup failure; the networkPlugin cni text is its detail.
• KubeletNotReady: container runtime network not ready: NetworkPluginNotReady — the node-level symptom of an unready CNI; see the node NotReady guide.
• Pod stuck in ContainerCreating — the generic state a CNI failure produces.
• no IP addresses available in range set — the IPAM-exhaustion variant covered above.

Frequently Asked Questions

Why is the pod stuck in ContainerCreating instead of crashing? The container never starts because its network sandbox could not be built. The kubelet keeps retrying sandbox creation on a backoff, so the pod sits in ContainerCreating with 0/1 ready and produces no container logs until the CNI succeeds.

What does failed to find plugin "calico" in path [/opt/cni/bin] mean exactly? The CNI config in /etc/cni/net.d references a plugin type (calico) whose binary is not present in /opt/cni/bin. Usually the CNI DaemonSet did not run on the node or was removed; reinstalling/restarting it copies the binaries back.

How do I fix cni0 already has an IP address different from ...? The host bridge cni0 kept an address from a previous pod CIDR. Delete the bridge (ip link delete cni0) and restart the kubelet so the CNI recreates it with the node’s current CIDR. Do this per affected node.

The CNI pods are Running but pods still fail — what now? Check the CNI pod logs for datastore errors (connection is unauthorized/refused) and confirm the node has a spec.podCIDR. “Running” is not “Ready” for the datastore connection; a brief kubectl delete pod of the CNI agent often re-establishes it.