🚨 Kubernetes Error Guide: 'connect: connection refused' Pod-to-Pod Networking

Exact Error Message

When one pod tries to open a TCP connection to another pod or a Service and the target host is reachable but nothing is accepting on that port, the connection is actively refused:

dial tcp 10.244.2.17:8080: connect: connection refused

The same condition shows up in HTTP clients, curl, and language runtimes:

Get "http://payments.default.svc.cluster.local:8080/healthz": dial tcp 10.96.41.7:8080: connect: connection refused
curl: (7) Failed to connect to payments port 8080: Connection refused
Error: connect ECONNREFUSED 10.244.2.17:8080

Note: this is in-cluster application traffic. It is not the kubectl control-plane error The connection to the server was refused, which is about reaching the API server.

What the Error Means

connection refused is a specific, informative failure. The client’s TCP SYN reached the destination IP, and the destination host responded with a TCP RST — meaning the host is up and routable, but no process is listening on that port. This is fundamentally different from a timeout (i/o timeout/connection timed out), where packets are silently dropped by a missing route, firewall, or NetworkPolicy, and the client gets no answer at all.

Because the host answered (with a refusal), the network path is largely working. The problem is at the destination: the application is not listening on the expected port, the Service is forwarding to the wrong targetPort, the pod is still starting, or the pod that was serving has crashed. When the destination is a Service IP, kube-proxy DNAT picks a backend pod; if that pod is not listening, you get the same refusal. Diagnosis is about confirming what is actually listening where, not chasing routing.

Common Causes

• App not listening yet / crashed — the container is starting, crashlooping, or bound but exited; nothing holds the port.
• Wrong listen address — the app binds 127.0.0.1 instead of 0.0.0.0, so it refuses connections from other pods.
• Service targetPort mismatch — the Service port is right but targetPort does not match the container’s real port.
• Wrong container port — traffic is sent to a port the app does not use.
• No/failing readiness probe — without readiness gating, the Service routes to a pod before it is listening.
• Selector routes to wrong/extra pods — the Service selector matches pods that do not serve that port.
• Sidecar/init ordering — a sidecar proxy (or the app behind it) is not up when traffic arrives.

How to Reproduce the Error

Deploy an app listening on 8080 but expose a Service pointing at the wrong targetPort:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 1
  selector: { matchLabels: { app: web } }
  template:
    metadata: { labels: { app: web } }
    spec:
      containers:
        - name: web
          image: hashicorp/http-echo
          args: ["-listen=:8080", "-text=hi"]
          ports: [{ containerPort: 8080 }]
---
apiVersion: v1
kind: Service
metadata: { name: web }
spec:
  selector: { app: web }
  ports:
    - port: 80
      targetPort: 9090   # WRONG: app listens on 8080

kubectl apply -f web.yaml
kubectl run client --rm -it --image=busybox -- wget -qO- http://web:80

wget: can't connect to remote host (10.96.x.x): Connection refused

The Service forwards port 80 to targetPort: 9090, but the app listens on 8080, so the backend refuses.

Diagnostic Commands

# Is the target pod Running and Ready? (Ready gates Service membership)
kubectl get pods -o wide -l app=web

# Does the Service have endpoints, and on which port?
kubectl get endpoints web -o wide
kubectl get endpointslices -l kubernetes.io/service-name=web -o yaml | grep -A3 ports

# Compare Service targetPort against the container's real port
kubectl get svc web -o jsonpath='{.spec.ports[*]}{"\n"}'
kubectl get pod <pod> -o jsonpath='{.spec.containers[*].ports}{"\n"}'

# Hit the pod IP directly, bypassing the Service, to isolate app vs Service
kubectl run probe --rm -it --image=busybox -- wget -qO- http://<podIP>:8080

# Confirm what the container is actually listening on
kubectl exec <pod> -- sh -c 'netstat -ltn 2>/dev/null || ss -ltn'

# Is the pod crashlooping (nothing listening)?
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].restartCount}{"\n"}'

Hitting the pod IP directly is the key split: if the pod IP refuses too, the app is the problem; if only the Service refuses, it is a targetPort/selector mismatch.

Step-by-Step Resolution

1. Confirm refused, not timeout. connection refused (host answered) points at the destination app/port. A timeout points at routing/NetworkPolicy instead — a different fix. Read the exact error word.

2. Test the pod IP directly. Bypass the Service:

kubectl run probe --rm -it --image=busybox -- wget -qO- http://<podIP>:<containerPort>

If the pod IP also refuses, the application is not listening — go to step 3. If the pod IP works but the Service refuses, go to step 4.

3. Fix the application listener. Confirm what the container listens on with netstat/ss inside the pod. Two common faults: the app crashed/crashlooped (check restartCount and logs), or it bound 127.0.0.1 instead of 0.0.0.0 (only loopback works, other pods are refused). Fix the bind address to 0.0.0.0 or the crash cause, and ensure the port matches.

4. Fix Service targetPort/selector. Compare the Service targetPort to the container port and align them:

kubectl get svc web -o jsonpath='{.spec.ports[*]}{"\n"}'
kubectl get pod <pod> -o jsonpath='{.spec.containers[*].ports}{"\n"}'

Set targetPort to the real container port. Verify the selector matches only pods that serve that port, and that kubectl get endpoints web lists the expected pod IPs.

5. Add or fix readiness. If the refusal is intermittent during rollouts, the Service is routing to pods before they listen. Add a readinessProbe on the serving port so pods join endpoints only when ready:

readinessProbe:
  tcpSocket: { port: 8080 }
  initialDelaySeconds: 3

6. Check sidecar ordering. With a service mesh or sidecar proxy, ensure the proxy and app are both up before traffic arrives; use startup/readiness gating so the Service does not route prematurely.

Prevention and Best Practices

• Always define a readinessProbe on the serving port so pods receive traffic only when actually listening — this prevents most rollout-time refusals.
• Bind application servers to 0.0.0.0, never 127.0.0.1, inside containers.
• Use named ports and reference them by name in the Service targetPort so port numbers cannot drift apart.
• Verify kubectl get endpoints <svc> shows the expected pod IPs after every deploy; an empty or wrong endpoints list predicts refusals.
• Distinguish refused (app/port) from timeout (routing/NetworkPolicy) in runbooks so responders chase the right layer.
• Keep startup, readiness, and liveness probes consistent with the real listen port for sidecar and mesh setups. More patterns in the Kubernetes & Helm guides.

Related Errors

• The connection to the server was refused — the kubectl-to-apiserver refusal, a different layer entirely.
• No endpoints available for service — the Service has zero ready backends, a closely related routing failure.
• SERVFAIL from CoreDNS — a name-resolution failure that happens before any connection attempt.

Frequently Asked Questions

How is connection refused different from a timeout? Refused means the destination host answered with a TCP RST — it is up but nothing is listening on that port. A timeout means packets were silently dropped (missing route, firewall, or NetworkPolicy) and no answer came back. Refused points at the app/port; timeout points at the network path.

The pod is Running but connections are refused. Why? Running only means the container process started, not that it is listening or ready. Check Ready status, the readinessProbe, and what the process actually binds with ss -ltn inside the pod. An app can be Running while still initializing or bound to the wrong address.

Connecting to the pod IP works but the Service refuses. What is wrong? The Service targetPort does not match the container’s real port, or the selector routes to pods that do not serve it. Compare kubectl get svc -o jsonpath='{.spec.ports[*]}' with the container ports, and verify kubectl get endpoints lists the right IPs.

Why does this only happen during deployments? Without a readiness probe, the Service adds new pods to its endpoints before the app is listening, so early connections are refused until the process binds. Add a readinessProbe on the serving port to gate membership.

The app works locally but refuses inside the cluster. It is almost certainly binding 127.0.0.1. Loopback works for same-pod traffic but refuses connections from other pods. Bind to 0.0.0.0 so the container accepts cluster traffic.