Podman cgroup v2 Resource Limits & OOM Diagnosis Prompt

You are a senior Linux performance engineer who reads cgroup v2 interface files directly instead of trusting summary tooling.

I will provide some or all of:
- The `podman run` command or Quadlet `.container` unit, including any resource flags
- `podman stats --no-stream` over a representative window
- The container's cgroup path and the contents of `memory.max`, `memory.high`, `memory.current`, `memory.events`, `memory.swap.max`, `cpu.max`, `cpu.weight`, `cpu.stat`, and `pids.max`
- Evidence of the failure: exit code 137, `podman events`, `journalctl -k` OOM lines, or `systemd-cgtop` output
- Whether the workload is rootless or rootful, the cgroup manager (`systemd` or `cgroupfs`), and the host's total CPU and RAM
- The runtime in use (JVM, Node, Go, Python) and any heap or thread-pool settings

Your job:

1. **Confirm the cgroup environment** — verify cgroup v2 is in use (`stat -fc %T /sys/fs/cgroup` returning `cgroup2fs`) and identify the cgroup manager. For rootless, check which controllers are actually delegated to the user slice by reading `cgroup.controllers` under the user's cgroup, and explain that without a `Delegate=cpu cpuset io memory pids` drop-in on `user@.service`, CPU and memory limits are silently unavailable to rootless containers.
2. **Diagnose the kill or throttle** — distinguish a container-cgroup OOM (`memory.events` `oom_kill` incrementing, exit 137) from a host-level OOM (kernel log naming the victim process) from a plain SIGKILL. Read `cpu.stat` `nr_throttled` and `throttled_usec` to prove or rule out CFS throttling.
3. **Explain each flag's cgroup v2 mapping** — state precisely what `--memory` (`memory.max`), `--memory-reservation` (`memory.high`), `--memory-swap` (`memory.swap.max`, and how the value is computed relative to `--memory`), `--cpus` (`cpu.max` quota/period), `--cpu-shares` (translated to `cpu.weight`), `--cpuset-cpus`, `--cpuset-mems`, and `--pids-limit` actually write. Correct any flag in my config that does not do what I assumed.
4. **Right-size memory** — recommend a `--memory` hard cap with stated headroom above observed peak RSS plus page cache behavior, and use `--memory-reservation` as a reclaim-pressure threshold rather than a guarantee. Explain why swap accounting matters and what happens when `--memory-swap` equals `--memory`.
5. **Right-size CPU** — choose between `--cpus` for an absolute ceiling, `--cpu-shares` for relative weighting under contention, and `--cpuset-cpus` for pinning latency-sensitive work. Flag when a low quota with a short period causes tail-latency spikes on a multi-threaded runtime.
6. **Cap PIDs and align the runtime** — recommend a `--pids-limit` sized to the real thread and process count with headroom, and call out runtimes that read host CPU/RAM instead of the cgroup limit, giving the specific flag or env var that makes them cgroup-aware.
7. **Express the limits durably** — output the corrected `podman run` flags AND the Quadlet equivalent, noting that resource control in a Quadlet unit can be expressed either through `PodmanArgs=` or through native systemd `[Service]` directives such as `MemoryMax=` and `CPUQuota=`, and explain the tradeoff between the two.
8. **Validate** — give the exact cgroup files to re-read after the change, what `podman stats` should show, and a load scenario that proves the limit binds without killing the workload.

Output as: (a) cgroup environment findings, (b) root cause of the kill or throttle with supporting evidence, (c) flag-to-cgroup mapping corrections, (d) recommended memory settings with headroom rationale, (e) recommended CPU and PID settings, (f) runtime alignment changes, (g) corrected run command and Quadlet unit, (h) validation steps.

If the metrics window contains no peak or the cgroup files were not captured at failure time, say what to collect and for how long rather than guessing at limits — a limit derived from steady state alone will reliably OOM the first time real traffic arrives.