☸️ Kubernetes Error Guide: 'Pod sandbox changed, it will be killed and re-created' CNI Churn

Overview

Warning  SandboxChanged  2m10s (x14 over 6m)  kubelet
  Pod sandbox changed, it will be killed and re-created.

Every Kubernetes pod runs inside a sandbox — a pause container that owns the pod’s Linux network namespace, cgroup, and IPC. Your application containers join that sandbox’s namespace. When the kubelet notices the sandbox is gone or unhealthy, it emits Pod sandbox changed, it will be killed and re-created and rebuilds the whole pod: it tears down the old sandbox, asks the CNI plugin for a fresh network setup, and restarts every container.

Seeing this once is normal (a pod starting or a node rebooting). Seeing it repeat — x14 over 6m — is a symptom of sandbox churn: something keeps destroying or invalidating the sandbox, so the pod restarts in a loop and never stabilizes. The root cause is almost always the container runtime or the CNI network plugin, not your application.

Symptoms

• Pods cycle between ContainerCreating and Running, with climbing RESTARTS.
• kubectl describe pod shows repeated SandboxChanged warnings.
• kubectl get events is flooded with FailedCreatePodSandBox alongside SandboxChanged.
• Application containers log clean startups but get killed seconds later with no crash.
• The problem clusters on one node or appears fleet-wide right after a CNI upgrade.

Common Root Causes

1. The CNI plugin is failing to set up networking

Calico, Cilium, Flannel, or the AWS VPC CNI cannot allocate an IP or program routes, so sandbox creation fails and the kubelet retries. Look for failed to setup network for sandbox in the events.

2. The container runtime is restarting or garbage-collecting sandboxes

A containerd/crio restart, or aggressive image/container garbage collection, kills pause containers out from under running pods.

3. The pause container image cannot be pulled

If the pause (sandbox) image is missing and the registry is unreachable, the runtime cannot recreate the sandbox and loops.

4. Node resource pressure

Memory or PID pressure causes the kubelet or runtime to kill sandbox containers, or the OOM killer reaps the pause process.

5. A crashing init/main container that keeps failing readiness

When the primary container repeatedly exits, the kubelet may recreate the sandbox as part of the restart backoff, producing the same event.

Diagnostic Workflow

Step 1: Read the pod events in full

kubectl describe pod <pod> -n <namespace> | sed -n '/Events:/,$p'
kubectl get events -n <namespace> --sort-by=.lastTimestamp | grep -i sandbox

Note whether SandboxChanged is paired with FailedCreatePodSandBox and a CNI message.

Step 2: Check CNI and node health

kubectl get pods -n kube-system -o wide | grep -Ei "calico|cilium|flannel|aws-node|weave"
kubectl describe node <node> | grep -A5 Conditions

Step 3: Inspect the runtime on the affected node

sudo crictl pods
sudo crictl ps -a
sudo journalctl -u containerd -n 200 --no-pager | grep -i "sandbox\|cni"

Step 4: Confirm the pause image is present

sudo crictl images | grep pause
sudo crictl info | grep -i sandboxImage

Step 5: Look for resource pressure and OOM kills

kubectl top node <node>
sudo dmesg -T | grep -i "oom\|killed process"

Step-by-Step Resolution

1. Fix the CNI first — it is the top cause. Confirm the CNI DaemonSet is healthy and restart the agent on the affected node:

   kubectl -n kube-system rollout restart daemonset <cni-daemonset>
   kubectl -n kube-system get pods -o wide | grep <node>

2. Pre-pull or fix the pause image if crictl images lacks it:

   sudo crictl pull registry.k8s.io/pause:3.9

   Ensure sandboxImage in /etc/containerd/config.toml matches a pullable tag, then sudo systemctl restart containerd.

3. Relieve node pressure. If OOM kills are hitting sandboxes, evict low-priority pods and add memory requests/limits so scheduling reflects real usage:

   kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

4. Restart the container runtime on the node if garbage collection or a stuck daemon is churning sandboxes:

   sudo systemctl restart containerd

5. Delete the stuck pod to force a clean recreate once the underlying cause is fixed:

   kubectl delete pod <pod> -n <namespace>

6. Verify stability — the restart count should stop climbing:

   kubectl get pod <pod> -n <namespace> -w

Prevention

• Keep CNI plugins on tested, pinned versions and roll upgrades node-by-node; a bad CNI rollout is the fastest way to trigger fleet-wide sandbox churn.
• Set realistic memory requests and limits so the scheduler avoids over-packing nodes into OOM territory.
• Mirror the pause image into your private registry so sandbox creation never depends on a flaky upstream pull.
• Alert on FailedCreatePodSandBox and SandboxChanged event rates, not just pod restarts.
• Drain nodes before runtime or kernel upgrades so sandbox teardown is graceful.

Related Errors

• Failed to create pod sandbox: ... failed to setup network — the specific CNI failure that usually drives this event.
• network: no IP addresses available in range — IPAM exhaustion, a common CNI cause of repeated sandbox failures.
• container runtime network not ready: cni config uninitialized — the CNI never initialized on the node.
• Back-off restarting failed container — an application crash loop, which can co-occur but is a different root cause.

Frequently Asked Questions

Is Pod sandbox changed always a problem? No. A single occurrence when a pod starts, a node reboots, or the runtime restarts is normal. It is only a concern when it repeats and the pod cannot stabilize.

How do I tell if it is CNI or my application? Check whether SandboxChanged is paired with FailedCreatePodSandBox and a network setup error — that points at CNI. If your main container is CrashLoopBackOff with a nonzero exit code, the application is the driver.

Which log has the real cause? The container runtime log on the node (journalctl -u containerd) plus kubectl describe pod events. The CNI plugin’s message there names the exact failure — IP exhaustion, missing route, or a timeout.

Can I just delete the pod to fix it? Deleting forces a clean recreate, but if the CNI or runtime is still broken the new pod loops the same way. Fix the node-level cause first, then delete. Generate targeted diagnostics with the DevOps AI prompt library.

Where can I learn more? See the Kubernetes & Helm guides.