🚨 Docker Error Guide: 'health: starting → unhealthy' — Fix a Failing HEALTHCHECK

Overview

A container’s HEALTHCHECK command keeps returning a non-zero exit code, so after the configured number of retries Docker flips the container’s status from health: starting to (unhealthy). You see it in docker ps:

CONTAINER ID   IMAGE        STATUS                     PORTS      NAMES
a1b2c3d4e5f6   myapp:1.4.2  Up 2 minutes (unhealthy)   8080/tcp   web-1

An unhealthy container is still running — Docker does not restart or kill it by default. But orchestrators (Swarm, Compose depends_on: condition: service_healthy, load balancers) treat it as a failed dependency and refuse to route traffic or start downstream services.

Symptoms

• docker ps shows (unhealthy) or is stuck on (health: starting) and never becomes healthy.
• depends_on with condition: service_healthy blocks other services from starting.
• The app answers requests fine when you curl it manually, yet Docker still marks it unhealthy.
• Swarm keeps rescheduling the task; a load balancer removes the backend.

Common Root Causes

• App not ready yet, start-period too short — the check begins before the app finishes booting (DB migrations, JIT warmup), so early failures count against retries.
• Wrong check target — probing localhost when the app binds only to 0.0.0.0 on a different port, or checking / when only /healthz exists.
• Missing tool in the image — the healthcheck calls curl or wget, but the slim/distroless base image doesn’t include it, so the command exits 127.
• Timeout too aggressive — a slow endpoint exceeds timeout, so the check is killed and counted as a failure.
• Check tests the wrong thing — it validates a dependency (database) that is down, not the container itself.

Diagnostic Workflow

Start with the health status and the recorded probe output:

docker ps --format 'table {{.Names}}\t{{.Status}}'
docker inspect --format '{{json .State.Health}}' web-1 | jq

The .State.Health.Log array holds the last few probes with their ExitCode, Start, End, and captured Output — this is usually the whole story.

Read the HEALTHCHECK instruction baked into the image (interval / timeout / retries / start-period):

docker inspect --format '{{json .Config.Healthcheck}}' myapp:1.4.2 | jq

Run the exact healthcheck command manually inside the container to see the real error:

docker exec web-1 curl -fsS http://localhost:8080/healthz
docker exec web-1 sh -c 'wget -qO- http://localhost:8080/healthz || echo FAILED'

If the tool is missing you’ll get exec: "curl": executable file not found or exit 127. Confirm what the app actually binds to:

docker exec web-1 sh -c 'netstat -tlnp 2>/dev/null || ss -tlnp'

Watch health transitions live to correlate with app startup:

docker events --filter event=health_status --filter container=web-1

Example Root Cause Analysis

A team ships myapp:1.4.2 with this Dockerfile healthcheck:

HEALTHCHECK --interval=10s --timeout=2s --retries=3 \
  CMD curl -fsS http://localhost:8080/ || exit 1

docker ps shows web-1 as (unhealthy). docker inspect --format '{{json .State.Health}}' web-1 | jq reveals every log entry has "ExitCode": 22 with output curl: (22) The requested URL returned error: 404.

Running it manually confirms the app is up but / returns 404 — the readiness route is /healthz. The check was pointed at the wrong path. The fix is a one-line change:

HEALTHCHECK --interval=10s --timeout=5s --retries=3 --start-period=30s \
  CMD curl -fsS http://localhost:8080/healthz || exit 1

Adding --start-period=30s also stops the slow first boot (migrations) from burning retries, and widening --timeout to 5s absorbs a cold-cache spike. After rebuilding, web-1 reaches (healthy) within one interval.

Prevention Best Practices

• Point the check at a dedicated, dependency-free endpoint like /healthz that returns 200 only when the app can serve traffic.
• Always set --start-period to cover realistic boot time so startup failures don’t count against retries.
• Prefer a language-native check or wget on slim images rather than assuming curl exists — or install the tool explicitly.
• Keep --timeout comfortably above your p99 latency for the health route.
• Distinguish liveness (is the process up) from readiness (can it serve) — don’t fail health just because a downstream is briefly unavailable, unless that’s intended.
• Validate the Dockerfile before shipping with the Dockerfile validator.

Quick Command Reference

# Current health status
docker ps --format 'table {{.Names}}\t{{.Status}}'

# Full health record with probe log
docker inspect --format '{{json .State.Health}}' web-1 | jq

# The configured HEALTHCHECK
docker inspect --format '{{json .Config.Healthcheck}}' myapp:1.4.2 | jq

# Run the check by hand
docker exec web-1 curl -fsS http://localhost:8080/healthz

# What the app is actually listening on
docker exec web-1 sh -c 'ss -tlnp'

# Watch health transitions
docker events --filter event=health_status --filter container=web-1

Conclusion

An (unhealthy) container almost always means the healthcheck, not the app, is misconfigured — a wrong path, a missing tool, a timeout that is too tight, or a start-period that is too short. Let .State.Health.Log tell you the exact exit code and output, reproduce the command with docker exec, then align the HEALTHCHECK interval, timeout, retries, and start-period with how your app really behaves. For more runtime fixes see the Docker guides.