🦭 Podman Error: Container Stuck 'unhealthy' or Healthcheck Never Firing

Exact Error Message

$ podman ps
CONTAINER ID  IMAGE                       COMMAND     STATUS                        NAMES
7c1e4b90a2df  registry.example.com/api:3  /entry.sh   Up 4 minutes (unhealthy)      api

Inspecting the health state shows the failing probe output:

$ podman inspect --format '{{json .State.Health}}' api
{"Status":"unhealthy","FailingStreak":3,"Log":[{"Start":"2026-07-19T10:02:11Z",
"End":"2026-07-19T10:02:11Z","ExitCode":127,
"Output":"OCI runtime exec failed: exec failed: unable to start container process: exec: \"curl\": executable file not found in $PATH"}]}

Or the opposite symptom — a container that never leaves starting because the check never runs at all:

$ podman inspect --format '{{.State.Health.Status}}' api
starting

What It Means

Podman has no daemon, so it cannot run a background loop that polls container health the way Docker does. Instead, when you start a container with a healthcheck, Podman creates a transient systemd timer (plus a matching service) that invokes podman healthcheck run <container> on your configured interval. The result is written back into the container’s State.Health record, and after --health-retries consecutive non-zero exits the status flips from starting to unhealthy.

That architecture produces two distinct failure modes that look similar in podman ps but have nothing in common. The first is a genuinely failing probe: the command runs and exits non-zero, usually because the tool it invokes is absent from a minimal image, the shell-form quoting is wrong, or the app truly is not ready. The second is a probe that never executes because the transient timer was never created or the user’s systemd manager is gone — the classic rootless case where you log out and the whole user session, timers included, is torn down. In that state health simply freezes at its last value forever, most often starting.

Common Causes

• The probe command depends on curl or wget, which are absent from distroless, Alpine-minimal, or scratch-based images.
• Shell-form quoting is wrong, so the whole string is treated as one binary name and exits 127.
• No user session or lingering is enabled, so rootless transient timers never run — the same root cause as crun sd-bus transport endpoint.
• --health-start-period is too short, so retries are exhausted before the application finishes booting.
• A HEALTHCHECK inherited from the base image is probing an endpoint your derived image does not serve.
• The probe exits 2 or another non-zero code expecting “warning” semantics; Podman treats anything non-zero as unhealthy.

Diagnostic Commands

Read the full health record, including the rolling log of recent probe attempts and their exit codes:

podman inspect --format '{{json .State.Health}}' api | python3 -m json.tool
podman inspect --format '{{.State.Health.FailingStreak}}' api

Check what healthcheck configuration the container actually got — this reveals a HEALTHCHECK silently inherited from the image:

podman inspect --format '{{json .Config.Healthcheck}}' api
podman image inspect --format '{{json .Config.Healthcheck}}' registry.example.com/api:3

Run the probe on demand. This bypasses the timer entirely and tells you whether the command or the scheduling is broken:

podman healthcheck run api
echo "exit=$?"

Confirm the transient timer exists. If nothing matching the container appears here, the check is not scheduled and the command is not your problem:

systemctl --user list-timers --all | grep -i api
systemctl --user list-units --all '*healthcheck*'

Verify the probe binary exists inside the image before blaming the application:

podman exec api sh -c 'command -v curl wget; echo "---"; ls /bin /usr/bin | head -40'

Read the probe’s own output from the journal, where the transient service logs land:

journalctl --user -u '*healthcheck*' --since '15 min ago' --no-pager
journalctl --user -t podman --since '15 min ago' --no-pager | grep -i health

Step-by-Step Resolution

1. Determine which failure mode you have. Run the probe manually — if it succeeds by hand but health never updates, the timer is the problem, not the command:

podman healthcheck run api && echo "probe OK, suspect the timer"

2. If the timer is missing under rootless Podman, enable lingering so your systemd user manager and its transient timers survive logout:

loginctl enable-linger "$USER"
loginctl show-user "$USER" --property=Linger
systemctl --user list-timers --all | grep -i api

3. If the probe exits 127, the binary is missing. Either install it in the image or use a check that needs no extra tooling:

# Option A: install the tool
RUN microdnf install -y curl && microdnf clean all

# Option B: no external binary required
HEALTHCHECK --interval=15s --timeout=3s --retries=3 --start-period=30s \
  CMD ["/app/bin/api", "--health-probe"]

4. Fix shell-form quoting. Exec form takes an argv array; shell form needs a real shell present in the image to interpret pipes and redirects:

# Wrong: entire string treated as one executable
podman run -d --name api --health-cmd "curl -f http://localhost:8080/healthz" registry.example.com/api:3

# Right: explicit shell, single quoted argument
podman run -d --name api \
  --health-cmd 'CMD-SHELL curl -fsS http://localhost:8080/healthz || exit 1' \
  --health-interval=15s \
  --health-retries=3 \
  --health-timeout=3s \
  --health-start-period=30s \
  --health-on-failure=kill \
  registry.example.com/api:3

5. Widen the start period if the app is simply slow to boot. During --health-start-period failures do not count toward the retry budget:

podman run -d --name api \
  --health-cmd 'CMD-SHELL wget -qO- http://localhost:8080/healthz >/dev/null || exit 1' \
  --health-start-period=90s --health-interval=20s --health-retries=5 \
  registry.example.com/api:3

6. Encode the working probe in a Quadlet unit so it is reproducible across reboots:

# ~/.config/containers/systemd/api.container
[Container]
Image=registry.example.com/api:3
PublishPort=8080:8080
HealthCmd=CMD-SHELL curl -fsS http://localhost:8080/healthz || exit 1
HealthInterval=15s
HealthRetries=3
HealthTimeout=3s
HealthStartPeriod=30s
HealthOnFailure=kill

[Install]
WantedBy=default.target

systemctl --user daemon-reload
systemctl --user start api.service
podman inspect --format '{{.State.Health.Status}}' api

For drafting probe commands and Quadlet health blocks against a specific service, the Podman prompts in the prompt library can generate a reviewed configuration.

Prevention

• Always pair --health-cmd with an explicit --health-start-period sized to your slowest cold start.
• Prefer probes that use a binary already shipped in the image over adding curl to a minimal base.
• Enable loginctl enable-linger on every host running rootless workloads with healthchecks.
• Treat exit codes as binary: 0 healthy, anything else unhealthy — never rely on a “warning” code.
• Declare healthchecks in Quadlet Health*= keys rather than ad hoc podman run flags so they survive reboots.
• Check podman image inspect for an inherited HEALTHCHECK whenever you adopt a new base image.

Related Errors

• OCI runtime exec failed: exec: "curl": executable file not found in $PATH — the probe binary is missing from the image, exit code 127.
• sd-bus call: Transport endpoint is not connected — no systemd user session, so timers never fire; see crun sd-bus transport endpoint.
• cgroup v1 is not supported — cgroup layout blocks the transient units Podman needs; see cgroup v1 not supported.
• unable to start container process — the container itself failed to start, so health never had a chance to run.

Frequently Asked Questions

Why does my healthcheck never run when I log out? Podman schedules healthchecks as transient systemd timers owned by your user manager. Logging out tears that manager down unless lingering is enabled. Run loginctl enable-linger $USER.

What exit codes does Podman accept? Only 0 means healthy. Every other exit code counts as a failed attempt, and after --health-retries consecutive failures the status becomes unhealthy. There is no intermediate “warning” code.

How do I test a probe without waiting for the interval? podman healthcheck run NAME executes the configured command immediately and records the result. Its shell exit status tells you whether the probe itself passed.

Where does the probe output go? Into State.Health.Log, readable with podman inspect --format '{{json .State.Health}}'. The transient service also logs to the journal, visible via journalctl --user. For more container fixes see the Podman guides.