☸️ Kubernetes Error Guide: 'rpc error: code = Unavailable desc = connection error' Kubelet to Runtime

Overview

E0716 10:22:14.882913    1421 remote_runtime.go:277] "ListPodSandbox with filter from runtime service failed"
  err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing:
  dial unix /run/containerd/containerd.sock: connect: connection refused\""

This message means the kubelet cannot talk to the container runtime over the CRI (Container Runtime Interface) gRPC socket. The kubelet is a gRPC client; containerd (or CRI-O) is the server listening on a Unix socket such as /run/containerd/containerd.sock. When that server is down, restarting, or the socket is missing, every runtime call — ListPodSandbox, PodSandboxStatus, RunPodSandbox — fails with code = Unavailable.

The Unavailable gRPC status code specifically signals a transport-level failure: the connection could not be established or was dropped. This is an infrastructure problem on the node, not a bad manifest or image. Until the socket is reachable again, the node cannot start, stop, or report pods, and it will usually flip to NotReady.

Symptoms

• The node shows NotReady in kubectl get nodes.
• Pods on the node stay Pending, ContainerCreating, or go Unknown.
• kubelet logs repeat rpc error: code = Unavailable desc = connection error with connection refused or no such file or directory.
• crictl commands hang or return the same Unavailable error.
• Node conditions report KubeletNotReady with container runtime is down or runtime network not ready.

Common Root Causes

1. The container runtime service is stopped or crashed

containerd or crio is not running — it crashed, was killed by an OOM event, or failed to start after a package upgrade. Nothing is listening on the socket, so the kubelet gets connection refused.

2. The runtime is restarting or mid-upgrade

During a containerd restart (a config reload, apt/yum upgrade, or automated patching), the socket briefly disappears. Transient Unavailable errors during this window are expected; persistent ones mean the restart failed.

3. Wrong or missing CRI socket path

The kubelet’s --container-runtime-endpoint (or containerRuntimeEndpoint in the config) points to a socket that does not exist — for example after migrating from Docker/dockershim to containerd, or from /var/run to /run.

4. A corrupted runtime state or bad config

An invalid /etc/containerd/config.toml (bad TOML, unsupported option, wrong SystemdCgroup setting) makes containerd exit on startup, so the socket never comes up.

5. Disk pressure or permission issues on the socket

A full /run tmpfs, or SELinux/AppArmor denials, can prevent the socket from being created or connected to even when the daemon appears to start.

Diagnostic Workflow

Step 1: Check node and kubelet status

kubectl get nodes -o wide
sudo systemctl status kubelet --no-pager
sudo journalctl -u kubelet -n 100 --no-pager | grep -i "Unavailable\|runtime"

Step 2: Check the runtime service directly

sudo systemctl status containerd --no-pager
sudo journalctl -u containerd -n 100 --no-pager

If you use CRI-O, substitute crio for containerd.

Step 3: Confirm the socket exists and responds

ls -l /run/containerd/containerd.sock
sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock version
sudo crictl info

A working runtime prints its version and RuntimeReady: true. A connection refused here reproduces the kubelet’s failure.

Step 4: Verify the kubelet’s configured endpoint

sudo grep -R "container-runtime-endpoint\|containerRuntimeEndpoint" /var/lib/kubelet/ /etc/systemd/system/kubelet.service.d/

Confirm the path matches the socket that actually exists in Step 3.

Step 5: Validate the runtime config

sudo containerd config dump | head -n 40

A parse error here explains a runtime that refuses to start.

Step-by-Step Resolution

1. Restart the container runtime. Most incidents clear once the daemon is back:

   sudo systemctl restart containerd
   sudo systemctl status containerd --no-pager

2. Confirm the socket is live with crictl before touching the kubelet:

   sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps

3. Fix the endpoint mismatch if the socket path is wrong. Point the kubelet at the real socket and reload:

   sudo sed -i 's#/var/run/dockershim.sock#/run/containerd/containerd.sock#' \
     /var/lib/kubelet/kubeadm-flags.env
   sudo systemctl daemon-reload
   sudo systemctl restart kubelet

4. Repair a broken config. If containerd config dump errors, regenerate a default and re-apply your customizations (cgroup driver, registry mirrors):

   sudo containerd config default | sudo tee /etc/containerd/config.toml
   sudo systemctl restart containerd

   Ensure SystemdCgroup = true if your kubelet uses the systemd cgroup driver.

5. Free disk space if /run or the root filesystem is full, then restart the runtime:

   df -h /run /var/lib/containerd

6. Restart the kubelet last and watch the node recover:

   sudo systemctl restart kubelet
   kubectl get nodes -w

The node should return to Ready within a minute once the runtime answers CRI calls.

Prevention

• Alert on the KubeletNotReady/RuntimeReady conditions and on the containerd/crio systemd unit so a down runtime pages you before pods stack up.
• Roll runtime upgrades node-by-node with kubectl drain first, so a restart never disrupts running workloads.
• Pin and test containerd config changes in staging; a bad config.toml is the most common self-inflicted cause.
• Keep the kubelet’s --container-runtime-endpoint in configuration management so it never drifts from the real socket path, especially after dockershim removal.
• Monitor node disk usage; a full /run or /var silently breaks the socket.

Related Errors

• container runtime network not ready: cni config uninitialized — the runtime is up but the CNI plugin is not, a different NotReady cause.
• validate service connection: CRI v1 runtime API is not implemented — a version/endpoint mismatch rather than a down socket.
• Failed to create pod sandbox: rpc error: code = Unknown — the runtime is reachable but sandbox creation itself failed.
• The connection to the server localhost:8080 was refused — an API-server/kubeconfig problem, not a CRI problem.

Frequently Asked Questions

What does code = Unavailable actually mean? It is a gRPC transport status meaning the client could not reach the server. For the kubelet, the server is the container runtime, so it almost always points to a down, restarting, or misconfigured CRI socket.

How do I know if it is containerd or the kubelet at fault? Run sudo crictl info directly. If crictl also fails with Unavailable, the runtime is the problem; if crictl works but the kubelet still errors, the kubelet’s endpoint path is wrong.

Is it safe to just restart containerd on a busy node? A restart is usually fast and containers keep running, but to be safe kubectl drain the node first during planned maintenance. For emergency recovery, restarting containerd then the kubelet is the standard fix.

Why did this start after upgrading Kubernetes? Dockershim removal in v1.24+ means the kubelet must point at containerd or CRI-O directly. A leftover dockershim.sock endpoint produces exactly this error. Generate the right resolution fast with the DevOps AI prompt library.

Where can I learn more? See the Kubernetes & Helm guides.