🚨 Kubernetes Error Guide: 'The connection to the server was refused' API Server Down

Exact Error Message

Every kubectl command fails immediately with a connection-refused error naming the API server host and port:

The connection to the server 10.0.0.10:6443 was refused - did you specify the right host or port?

You may also see it against a load balancer or localhost:

The connection to the server kube-api.internal:6443 was refused - did you specify the right host or port?

The connection to the server localhost:8080 was refused - did you specify the right host or port?

The third variant is a special case: localhost:8080 means kubectl found no kubeconfig at all and fell back to its legacy default, so it is really a missing-config problem rather than a down server.

What the Error Means

“Connection refused” is a TCP-level rejection: kubectl opened a socket to host:6443, and the operating system on the other end actively returned an RST because nothing is listening on that port. This is distinct from a timeout (context deadline exceeded) — refused means the host is reachable but the API server process is not accepting connections, or the client is pointed at the wrong place.

There are two broad explanations:

1. The API server is genuinely not listening — the kube-apiserver static pod crashed or never started, the kubelet that runs it is stopped, a load balancer in front of it is down, or a firewall rules out the path.
2. The client is pointed at the wrong endpoint — a stale or wrong server: URL in the kubeconfig, the wrong context, or no kubeconfig at all (the localhost:8080 case).

Common Causes

• kube-apiserver static pod in crashloop — the apiserver is a static pod managed by the kubelet from /etc/kubernetes/manifests. It crashloops when etcd is unhealthy, a flag is invalid, or its serving/client certificates have expired.
• Expired control-plane certificates — after one year, unrotated kubeadm certs expire and the apiserver fails to start or to talk to etcd, leaving 6443 closed.
• Wrong or stale kubeconfig — the server: URL points at an old IP, a decommissioned node, or the wrong cluster; or the active context selects a cluster that no longer exists.
• API server not listening / firewall / security group — the process is up but bound to a different address, or a host firewall, network policy, or cloud security group blocks 6443.
• Load balancer or VIP down — in HA control planes, clients reach the apiserver through an LB or keepalived VIP. If the LB has no healthy backends or the VIP failed over incorrectly, the connection is refused.
• localhost vs node IP — pointing kubeconfig at 127.0.0.1:6443 from a node where the apiserver binds a different interface, or relying on the legacy localhost:8080.
• kubelet not running / swap enabled — if the kubelet is down or refuses to start (for example swap is on and failSwapOn is true), it never launches the control-plane static pods, so the apiserver never comes up.

How to Reproduce the Error

On a kubeadm control-plane node, move the apiserver manifest out of the static-pod directory. The kubelet stops the pod within seconds and 6443 goes silent.

sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
# wait a few seconds for the kubelet to tear down the static pod
kubectl get nodes

The connection to the server 10.0.0.10:6443 was refused - did you specify the right host or port?

Restoring the manifest brings it back:

sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/

Diagnostic Commands

First decide whether the client is even pointed at the right place:

kubectl config view --minify
kubectl config current-context
kubectl config view -o jsonpath='{.clusters[0].cluster.server}'

https://10.0.0.10:6443

Probe the endpoint directly. A refused curl confirms nothing is listening; a successful one means the problem is your kubeconfig or auth:

curl -k --max-time 5 https://10.0.0.10:6443/healthz
nc -vz 10.0.0.10 6443

curl: (7) Failed to connect to 10.0.0.10 port 6443: Connection refused

On the control-plane node, inspect the static-pod containers and the kubelet that runs them:

sudo crictl ps -a | grep -E 'kube-apiserver|etcd'
sudo crictl logs $(sudo crictl ps -a --name kube-apiserver -q | head -1) 2>&1 | tail -30
sudo systemctl status kubelet
sudo journalctl -u kubelet --no-pager | tail -40
ls -l /etc/kubernetes/manifests/

CONTAINER   IMAGE   ... STATE    NAME             ATTEMPT
a1b2c3      ...      Exited   kube-apiserver   8

Check etcd health and certificate expiry, the two most common reasons the apiserver crashloops:

sudo kubeadm certs check-expiration
sudo crictl logs $(sudo crictl ps -a --name etcd -q | head -1) 2>&1 | tail -20

Step-by-Step Resolution

1. Confirm the target. If the error says localhost:8080, you have no kubeconfig loaded. Set it:

   export KUBECONFIG=/etc/kubernetes/admin.conf   # on a control-plane node
   # or for a workstation:
   mkdir -p ~/.kube && cp /etc/kubernetes/admin.conf ~/.kube/config

   If the server: URL is simply wrong or stale, switch context or fix the URL with kubectl config use-context <name>.

2. Test the endpoint with curl -k https://<host>:6443/healthz. If it responds, the server is up and your problem is the kubeconfig/context/auth — stop here. If it is refused, continue.

3. On the control-plane node, check the static pods. Run crictl ps -a for kube-apiserver. If it is Exited/crashlooping, read its logs. Typical findings:

   • open /etc/kubernetes/pki/apiserver.crt: ... certificate has expired → renew certs:

     sudo kubeadm certs renew all
     sudo systemctl restart kubelet

   • unknown flag / bad value → fix /etc/kubernetes/manifests/kube-apiserver.yaml (the kubelet reapplies it automatically).
   • connection refused to etcd 127.0.0.1:2379 → fix etcd first; the apiserver cannot start without it.

4. Verify the kubelet is running. If systemctl status kubelet shows it is dead, start it. If it refuses because swap is on:

   sudo swapoff -a
   sudo systemctl restart kubelet

   With the kubelet up, it relaunches the static pods from /etc/kubernetes/manifests.

5. Check the path in HA setups. If clients go through an LB/VIP, confirm the LB has healthy backends on 6443 and that the VIP is bound to a live node (ip addr, keepalived logs). Test each apiserver directly to isolate a single bad replica versus a dead LB.

6. Check firewall/security groups. Ensure 6443 is open from the client to the apiserver (host firewall, cloud security group, network policy). nc -vz <host> 6443 from the client confirms reachability.

Prevention and Best Practices

• Automate certificate renewal and alert at least 30 days before expiry; expired control-plane certs are a top cause of a dead apiserver. Run kubeadm certs check-expiration on a schedule.
• Run an HA control plane (3+ apiservers behind a load balancer) so a single node failure does not refuse all connections.
• Monitor the apiserver’s /healthz//readyz and the LB backend health externally, not just from inside the cluster.
• Keep admin.conf backed up and distribute kubeconfigs with the LB/VIP address, never a single node IP, so client config survives a node replacement.
• Ensure swap stays off on control-plane nodes and that the kubelet is enabled so static pods recover after a reboot.

Related Errors

• context deadline exceeded — the host is reachable but slow, rather than refusing the connection.
• Unable to connect to the server: dial tcp ... i/o timeout — the host is unreachable (firewall/routing), distinct from an active refusal.
• x509: certificate has expired or is not yet valid — the apiserver is reachable but rejects the TLS handshake.
• etcdserver: request timed out — an unhealthy etcd that keeps the apiserver from starting.

Frequently Asked Questions

What’s the difference between “connection refused” and “i/o timeout”? “Connection refused” means the host responded with a TCP reset — it is reachable but nothing is listening on 6443 (apiserver down, or wrong port). “i/o timeout” means no response at all — the host is unreachable, usually a firewall, routing, or down node. They point at different layers, so read the exact wording.

Why does kubectl suddenly say localhost:8080? That is the legacy default kubectl uses when it cannot find any kubeconfig. It usually means KUBECONFIG is unset, ~/.kube/config is missing, or a script cleared the environment. Set KUBECONFIG or restore ~/.kube/config and the real server URL returns.

The apiserver container keeps restarting — where do I look first? Read crictl logs for the kube-apiserver container. The first few lines almost always name the cause: an expired certificate, an invalid flag in the manifest, or a failed connection to etcd. Fix that root cause rather than restarting the kubelet repeatedly.

All my apiservers are up but clients still get refused — why? The load balancer or VIP in front of them is likely the problem. Check that the LB has healthy 6443 backends and that the VIP is bound to a live node. Test each apiserver IP directly; if those work but the LB address is refused, the fault is in the LB/VIP layer.

Can a stopped kubelet cause this on a control-plane node? Yes. On kubeadm clusters the apiserver runs as a static pod managed by the kubelet. If the kubelet is dead (crashed, or refusing to start because swap is enabled), it never launches the apiserver, so 6443 is closed. Restart the kubelet after fixing whatever stopped it.