Podman Healthcheck Design for systemd Prompt
Design container healthchecks that genuinely detect failure under Podman, including startup probes, on-failure actions, and correct interaction with systemd restart policy.
- Target user
- Platform engineers running Podman containers as systemd-managed services
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT
The prompt
You are a senior platform engineer who knows how Podman healthchecks actually execute. You know Podman implements them as transient systemd timers rather than a daemon loop, that `--health-startup-cmd` and its companion flags give you a separate startup probe so a slow boot does not count as failure, that `--health-on-failure` accepts `none`, `kill`, `restart`, or `stop`, and that `restart` interacts with the unit's own `Restart=` policy in ways that need thinking about. You also know most healthchecks in the wild check that a process is alive rather than that the service works, which is why they never fire.
I will describe the service. Design a healthcheck that detects real failure.
Steps:
1. **Define what unhealthy means for this service**: write the failure conditions the check must catch — a lost database connection, a full queue, a wedged worker pool — and reject checks like `curl localhost/` or a `pgrep` that pass while the service is useless.
2. **Design the probe command**: choose an endpoint or command that exercises the real dependency path but stays cheap enough to run on every interval, and confirm the tooling it needs actually exists in the image (a `curl`-based check fails silently on a minimal base with no curl).
3. **Separate startup from steady state**: use `--health-startup-cmd` with its own interval, retries, and timeout so a slow-starting service is not killed during boot, and set the steady-state `--health-interval`, `--health-timeout`, and `--health-retries` independently.
4. **Choose the failure action deliberately**: pick `--health-on-failure` from `none`, `kill`, `restart`, or `stop` based on whether restarting plausibly fixes the failure — restarting a container whose database is down produces a crash loop that adds load to an already-failing dependency.
5. **Reconcile with systemd restart policy**: describe how the chosen on-failure action combines with the unit's `Restart=` and rate limiting, and make sure the two cannot fight — a container systemd restarts and a healthcheck kills can loop indefinitely without either being obviously at fault.
6. **Connect it to auto-update**: note that a working healthcheck is what makes `podman auto-update` rollback possible, so a service that auto-updates needs a check that genuinely fails on a bad image, not one that passes as long as the process started.
7. **Set timeouts against reality**: base interval, timeout, and retries on the service's observed p99 response time and real startup duration, and show the arithmetic for how long a genuine outage takes to be detected.
8. **Express and verify**: give the Quadlet `HealthCmd=`, `HealthInterval=`, `HealthRetries=`, `HealthStartPeriod=`, `HealthOnFailure=`, and startup-probe directives, plus how to test with `podman healthcheck run <ctr>` and `podman inspect --format '{{.State.Health.Status}}'`, including how to force a failure to prove the action fires.
Fill in:
- Service and what it does: [DESCRIBE]
- Real failure modes seen in production: [LIST]
- Startup time, cold and warm: [DESCRIBE]
- External dependencies it needs: [LIST]
- Current healthcheck if any: [PASTE]
- Unit restart policy: [PASTE]
Output as: (a) the probe command with what it exercises and why it is cheap enough, (b) the full Quadlet health directives with the reasoning behind each timing value, and (c) the test procedure including how to force a failure and confirm the on-failure action and restart policy behave as intended.
Do not ship a healthcheck you have never seen fail. Force the failure in a test environment and confirm the action fires, otherwise you have added the appearance of monitoring without the substance.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Why this prompt works
Most container healthchecks are decorative. They check that a port is open or a process exists, which means they report healthy through exactly the outages that matter — the database connection pool is exhausted, the worker queue is wedged, requests are timing out, and the check keeps passing because the process is technically alive. Starting the prompt by demanding a written definition of what unhealthy means for this specific service, and explicitly rejecting liveness-only probes, is the difference between a check that detects failure and one that provides false comfort during it.
The startup-versus-steady-state separation solves the practical problem that pushes teams toward useless checks in the first place. A service that takes ninety seconds to warm caches needs a tolerant probe at boot and a strict one afterwards, and without a separate startup probe the only way to survive startup is to loosen the steady-state check until it detects nothing. Podman’s startup healthcheck flags exist precisely for this, and using them means the running check can be as strict as the service deserves.
The interaction between --health-on-failure and systemd’s Restart= is the part that most designs never consider, and it produces the ugliest failures. Two independent mechanisms both authorized to restart the container will happily loop against each other, and when the underlying cause is a failing external dependency, that loop directs a restart storm at a service already in trouble. Requiring the two policies to be reconciled explicitly, and closing with an instruction to force a real failure before shipping, is what turns a healthcheck from configuration that looks correct into behavior that has been observed working.
Related prompts
-
Podman systemd Auto-Update and Unit Hardening Prompt
Configure podman auto-update with safe rollback, correct restart policy, and systemd unit hardening for containers managed as Quadlet services.
-
docker-compose to Podman Quadlet Migration Prompt
Convert a docker-compose stack into Podman Quadlet systemd unit files with correct dependency ordering, networks, volumes, and rootless-safe port publishing.
-
Docker to Podman Team Migration Plan Prompt
Produce a phased Docker-to-Podman migration plan covering socket compatibility, CI impact, rootless gaps, and the developer workflow changes a team will actually hit.
-
Dockerfile to Buildah Build Script Conversion Prompt
Convert a Dockerfile into an equivalent scripted buildah build so image construction can use host tooling, secrets, and logic that Dockerfile syntax cannot express.
More Podman prompts & error guides
Browse every Podman prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.