☸️ Kubernetes Error Guide: 'Liveness probe failed: ... connection refused' Restart Loop

Overview

Warning  Unhealthy  31s (x6 over 2m)  kubelet
  Liveness probe failed: Get "http://10.244.3.14:8080/healthz": dial tcp 10.244.3.14:8080: connect: connection refused
Normal   Killing    31s (x2 over 90s)  kubelet
  Container app failed liveness probe, will be restarted

A liveness probe tells the kubelet whether a container is still healthy. When the probe fails enough consecutive times, the kubelet kills and restarts the container. connection refused means the kubelet reached the pod’s IP but nothing was listening on the target port — the TCP connection was actively rejected.

The trap is the restart loop: the app needs, say, 40 seconds to start listening, but the liveness probe begins checking after 10 seconds, fails, and the kubelet kills the container before it ever finishes booting. The container restarts, fails the probe again, and the pod lands in CrashLoopBackOff — even though the application code is fine. The fix is almost always probe configuration, not the app.

Symptoms

• kubectl get pod shows climbing RESTARTS and eventually CrashLoopBackOff.
• kubectl describe pod repeats Liveness probe failed: ... connection refused.
• Container logs show the app starting normally, then being terminated mid-startup.
• The app works fine when run locally or with kubectl port-forward once it is up.
• Slow-starting apps (JVM, Rails, large migrations) fail; fast ones on the same image pass.

Common Root Causes

1. The app is not listening yet when the probe starts

initialDelaySeconds is too short for the application’s real startup time, so the probe fires before the server binds its port.

2. Wrong probe port or path

The probe targets 8080 but the app listens on 3000, or hits /healthz when the route is /health. Both produce connection refused (wrong port) or non-200 responses (wrong path).

3. The app binds to localhost only

The server listens on 127.0.0.1 instead of 0.0.0.0, so the kubelet — connecting on the pod IP — is refused even though the process is up.

4. The health endpoint depends on a slow dependency

The /healthz handler blocks on a database or cache that is not ready, so it times out or refuses connections during startup.

5. No startupProbe for a genuinely slow app

Without a startupProbe, there is no separate grace period for boot, so the liveness probe and the slow start fight each other.

Diagnostic Workflow

Step 1: Read the events and probe config

kubectl describe pod <pod> -n <namespace> | sed -n '/Liveness:/,/Events:/p'
kubectl describe pod <pod> -n <namespace> | grep -A3 "Liveness probe failed"

Note the exact port and path the kubelet is hitting.

Step 2: Check restart count and last state

kubectl get pod <pod> -n <namespace> -o wide
kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.containerStatuses[0].lastState}{"\n"}'

Step 3: Confirm what the app actually listens on

kubectl logs <pod> -n <namespace> --previous | grep -i "listen\|started\|bound\|port"

Step 4: Test the endpoint from inside the pod

kubectl exec -it <pod> -n <namespace> -- sh -c "wget -qO- http://127.0.0.1:8080/healthz || echo REFUSED"

Step 5: Measure real startup time

kubectl logs <pod> -n <namespace> --timestamps | head

Compare the “server listening” timestamp against initialDelaySeconds.

Step-by-Step Resolution

1. Fix the port/path mismatch so the probe targets what the app serves:

   livenessProbe:
     httpGet:
       path: /health
       port: 3000
     initialDelaySeconds: 15
     periodSeconds: 10
     failureThreshold: 3

2. Add a startupProbe for slow starters. It gives the app up to failureThreshold * periodSeconds to boot, and liveness only begins after it passes:

   startupProbe:
     httpGet:
       path: /health
       port: 3000
     failureThreshold: 30
     periodSeconds: 5

   This example allows 150 seconds to start without loosening steady-state liveness.

3. Bind to all interfaces. In your app, listen on 0.0.0.0, not 127.0.0.1, so the kubelet can reach it on the pod IP.

4. Keep the health handler cheap. Make /health return 200 as soon as the process is up; move dependency checks into the readiness probe so a slow database does not kill the container.

5. Apply and roll out:

   kubectl apply -f deployment.yaml
   kubectl rollout status deployment/<name> -n <namespace>

6. Confirm restarts stop climbing:

   kubectl get pod <pod> -n <namespace> -w

Prevention

• Use a startupProbe for any app with variable or long boot times so liveness never races the startup.
• Keep liveness handlers dependency-free — liveness answers “is the process wedged?”, readiness answers “can it serve traffic yet?”.
• Set initialDelaySeconds and failureThreshold from measured startup time plus headroom, not guesses.
• Standardize on 0.0.0.0 binding and a documented health port across services.
• Test probe config in staging under realistic cold-start conditions before shipping.

Related Errors

• Readiness probe failed: connection refused — same cause, but it only removes the pod from Service endpoints instead of restarting it.
• Liveness probe failed: HTTP probe failed with statuscode: 500 — the app is listening but the health endpoint returns an error, not a connection refusal.
• Back-off restarting failed container — the CrashLoopBackOff that a failing liveness probe produces.
• context deadline exceeded in a probe — a timeout rather than a refused connection, usually a slow handler.

Frequently Asked Questions

What is the difference between connection refused and a probe timeout? connection refused means nothing is listening on the port — the app has not bound it yet or the port is wrong. A timeout (context deadline exceeded) means something is listening but responded too slowly. They point to different fixes.

Should I just increase initialDelaySeconds? It works but is blunt — it delays liveness for the whole pod lifetime. A startupProbe is better: it grants a generous boot window, then hands off to a tight liveness check.

Why does the same image pass sometimes and fail others? Variable startup time. Cold JVM warmup, migrations, or a slow dependency push boot past initialDelaySeconds intermittently. A startupProbe absorbs that variance.

Liveness or readiness — which should gate dependencies? Readiness. Put database and cache checks in the readiness probe so a dependency blip removes the pod from load balancing instead of restarting the container. Generate a tuned probe spec with the DevOps AI prompt library.

Where can I learn more? See the Kubernetes & Helm guides.