🚨 Kubernetes Error Guide: 'dial tcp <ip>:<port>: connect: no route to host'

Exact Error Message

A client tries to connect and the kernel immediately reports the destination as unreachable:

Error from server: error dialing backend: dial tcp 10.244.5.23:10250: connect: no route to host

upstream connect error or disconnect/reset before headers. reset reason: connection failure,
transport failure reason: delayed connect error: dial tcp 10.96.0.42:8080: connect: no route to host

Unlike a timeout, no route to host returns almost instantly. The kernel’s routing layer (or a firewall REJECT) actively decided the address cannot be reached, rather than waiting silently.

What the Error Means

no route to host is the user-space form of the kernel error EHOSTUNREACH. It is returned when the host has no usable route to the destination, or when something on the path answers with an ICMP “host unreachable” / a firewall REJECT. The key signal is speed: the error comes back fast because no waiting was involved — the stack already “knows” the address is unreachable.

In Kubernetes this most often appears when a connection targets a pod IP or Service that has gone stale. kube-proxy programs iptables (or IPVS) rules that DNAT Service ClusterIPs to live pod IPs; if those rules are stale, missing, or point at a pod on a node that has lost its route, you get an instant no route to host. Node-level routing or firewall changes produce the same symptom for cross-node pod traffic.

Common Causes

• Stale endpoints / pod IP reuse — the client cached a pod IP (or kube-proxy has a stale rule) for a pod that no longer exists; the IP now routes nowhere.
• kube-proxy not programming rules — kube-proxy is down or erroring, so Service ClusterIPs have no DNAT rules and resolve to a dead path.
• Firewall REJECT (not DROP) — a host firewall rule rejects with ICMP unreachable instead of silently dropping, yielding no route to host rather than a timeout.
• Node lost its pod route — a node’s route to another node’s pod CIDR was removed (CNI restart, route table change), so cross-node pod IPs are unreachable.
• Pod just terminated — connecting to a pod IP during/just after termination before endpoints updated.
• conntrack / stale NAT entries — a stale conntrack entry maps to a gone backend.

How to Reproduce the Error

Connect to a pod IP, then delete the pod and immediately reconnect to the same (now stale) IP:

kubectl run target --image=registry.k8s.io/e2e-test-images/agnhost:2.47 \
  --command -- /agnhost netexec --http-port=8080
POD_IP=$(kubectl get pod target -o jsonpath='{.status.podIP}')
kubectl delete pod target --wait=false

kubectl run probe --image=nicolaka/netshoot --rm -it --restart=Never -- \
  curl -m 5 http://$POD_IP:8080/

curl: (7) Failed to connect to 10.244.5.23 port 8080: No route to host

The route to the now-gone pod IP is removed and the connection fails instantly.

Diagnostic Commands

# Is the client using a stale endpoint? Compare current ready endpoints.
kubectl get endpoints <SERVICE> -n <NS>
kubectl get endpointslices -n <NS> -l kubernetes.io/service-name=<SERVICE>

# Confirm the target pod IP is current and the pod is Running
kubectl get pod -o wide -n <NS> | grep <DEST-IP>

# Is kube-proxy healthy on every node?
kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide
kubectl logs -n kube-system <KUBE-PROXY-POD> --tail=50

# Inspect routing and proxy rules on the node (read-only)
ip route show
iptables -t nat -L KUBE-SERVICES -n 2>/dev/null | grep <CLUSTERIP>

# CNI agent health (it programs the inter-node pod routes)
kubectl get pods -n kube-system -o wide | grep -E 'calico|cilium|flannel'

If the ClusterIP has no NAT rule, or the endpoint list does not match the IP your client used, you have found a stale-routing problem.

Step-by-Step Resolution

1. Confirm you are targeting a live address. The most common cause is a stale pod IP. Compare the IP the client used against the Service’s current Endpoints/EndpointSlices. If it is not in the list, the client cached a dead IP — connect through the Service name instead so kube-proxy picks a live backend.

2. Check kube-proxy. If Service ClusterIPs broadly return no route to host, kube-proxy may be down or failing to sync. Confirm its pods are Running on every node and the logs show successful rule syncs, not errors. Restart it on the affected node if it is stuck.

3. Look for a firewall REJECT. A firewall that REJECTs (rather than DROPs) produces an instant no route to host. Inspect host firewall rules and security groups for a reject targeting the pod CIDR or app port; change it to allow the needed traffic.

4. Restore inter-node pod routes. For cross-node failures, verify the node has a route to the destination pod CIDR. A CNI restart or manual route change can remove it. Confirm the CNI agent is healthy on both nodes so it can re-establish routes:

ip route show | grep <DEST-POD-CIDR>

5. Clear stale NAT/conntrack state if needed. Occasionally a stale conntrack entry pins traffic to a gone backend. Reconnecting through the Service and letting kube-proxy re-DNAT usually resolves it once endpoints are correct.

6. Re-test. After the fix, connect through the Service and confirm a fast, successful handshake:

kubectl run netshoot --image=nicolaka/netshoot --rm -it --restart=Never -- \
  curl -m 5 http://<SERVICE>.<NS>.svc.cluster.local:<PORT>/

Prevention and Best Practices

• Never hard-code pod IPs; always connect through Services/DNS so traffic follows live endpoints.
• Monitor kube-proxy health and rule-sync latency; stale or missing rules are a leading cause of this error.
• Prefer firewall DROP-with-intent or explicit allow rules, and know that REJECT surfaces as no route to host to clients.
• Keep CNI agents healthy so inter-node pod routes are continuously maintained.
• Set sensible readiness probes and terminationGracePeriodSeconds so endpoints drain before pods disappear, shrinking the stale-IP window. More in the Kubernetes & Helm guides.

Related Errors

• dial tcp i/o timeout — silent packet drops instead of an instant unreachable.
• connection to the server was refused — an active TCP reset from a listening host.
• no endpoints available for service — the Service has zero ready backends.
• lookup no such host on cluster.local — DNS resolution failing before any TCP attempt.

Frequently Asked Questions

Why does this fail instantly while i/o timeout hangs? no route to host means the kernel already has no route, or a firewall REJECT / ICMP unreachable answered immediately — there is nothing to wait for. A timeout means packets are silently dropped, so the stack waits the full deadline before giving up.

It works through the Service name but fails on a pod IP. Why? The pod IP you used is stale — that pod is gone and its route no longer exists. The Service name resolves to a current ClusterIP that kube-proxy DNATs to a live backend, which is exactly why you should connect by name.

All ClusterIPs suddenly return no route to host. What happened? Suspect kube-proxy: if it is down or failed to program iptables/IPVS rules on a node, Service VIPs have no path. Check kube-proxy pod health and logs on that node first.

Could a firewall cause this even though nothing changed in Kubernetes? Yes. A REJECT rule (host firewall, security group, or a security tool) added on the path returns ICMP unreachable, which clients see as no route to host. Audit firewall changes around the time it started.

Is restarting the node a fix? It can clear stale routes/conntrack and restart kube-proxy and the CNI, so it often “works” — but find the root cause (stale endpoints, kube-proxy, firewall) so it does not recur. Restarting indiscriminately just hides the pattern.