🦭 Podman Error: 'Exited (137)' Container Killed by SIGKILL

Exact Error Message

$ podman ps -a
CONTAINER ID  IMAGE                              COMMAND     CREATED         STATUS                     PORTS   NAMES
9f1c2a7b4e30  docker.io/library/openjdk:21-jdk   java -jar   4 minutes ago   Exited (137) 12 seconds ago        api

The same status shows up in podman inspect and in the logs of a systemd-managed container:

$ systemctl --user status container-api.service
● container-api.service - Podman container-api.service
     Active: failed (Result: exit-code) since Sun 2026-07-19 11:04:22 UTC
    Process: 41288 ExecStart=/usr/bin/podman start api (code=exited, status=137)

What It Means

Exit code 137 is not a Podman-specific error at all — it is the shell convention of 128 + signal number, and signal 9 is SIGKILL. The container’s main process (PID 1 inside the namespace) was killed outright, with no chance to run signal handlers, flush buffers, or shut down cleanly. Podman simply reports the wait status the OCI runtime handed back. The interesting question is never “what does 137 mean” but “who sent the SIGKILL”.

There are only two realistic senders. The first is the kernel’s out-of-memory killer, which fires when a cgroup’s memory usage hits its hard limit (memory.max under cgroup v2) and reclaim cannot free enough pages; the kernel picks a victim inside that cgroup and kills it. The second is something in userspace: podman kill, podman stop escalating to SIGKILL after its timeout expires, a systemd unit hitting TimeoutStopSec, a Kubernetes-style liveness supervisor, or an operator with a stray kill -9. Distinguishing the two takes about thirty seconds and completely changes the fix — an OOM-kill means resize the limit or the workload heap, while an external SIGKILL means your process is ignoring or being too slow to handle SIGTERM.

Common Causes

• The container exceeded its cgroup memory limit and the kernel OOM killer terminated it.
• The host itself ran out of memory and the global OOM killer picked the container’s process as the victim.
• podman stop sent SIGTERM, the process did not exit within the stop timeout, and Podman escalated to SIGKILL.
• A JVM or Node.js process sized its heap from host RAM rather than the container limit and grew past memory.max.
• A systemd unit managing the container hit TimeoutStopSec or OOMPolicy=kill and killed the whole control group.
• --memory was passed under rootless cgroup v1 where it is silently ignored, so the “limit” never existed and the host OOM killer fired instead.

Diagnostic Commands

The single most useful field tells you immediately whether the kernel did it:

podman inspect --format '{{.State.OOMKilled}} {{.State.ExitCode}} {{.State.Error}}' api

Watch the event stream, which records died events along with an explicit oom event when the kernel kills a container:

podman events --filter container=api --since 30m
podman events --filter event=oom --since 24h

Check the kernel log for the OOM report, which names the cgroup and the victim task:

journalctl -k --since "1 hour ago" | grep -iE "out of memory|oom-kill|killed process"

Confirm which cgroup version you are on — this determines whether rootless memory limits work at all:

podman info --format '{{.Host.CgroupsVersion}} {{.Host.CgroupManager}}'

For a running container, read the live cgroup counters. memory.events keeps a persistent tally of how many times the limit was hit:

CID=$(podman inspect --format '{{.Id}}' api)
cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/user.slice/libpod-$CID.scope/memory.max
cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/user.slice/libpod-$CID.scope/memory.events

Step-by-Step Resolution

1. Classify the kill. If OOMKilled is true, it is a memory problem; if it is false and the kernel log is silent, something in userspace sent the signal:

podman inspect --format '{{.State.OOMKilled}}' api
journalctl -k --since "1 hour ago" | grep -i "oom-kill" || echo "no kernel OOM - external SIGKILL"

2. If it was external, find the sender. A podman stop escalation is by far the most common: the default stop timeout is 10 seconds, and a process that does not handle SIGTERM will always be SIGKILLed at the end of it. Raise the timeout and make sure your entrypoint forwards signals:

podman stop --time 60 api
podman run -d --stop-timeout 60 --name api docker.io/library/openjdk:21-jdk java -jar /app.jar

Note that a shell-form entrypoint (sh -c "java -jar /app.jar") leaves the shell as PID 1 and swallows SIGTERM entirely; use exec form or --init so signals actually reach your process.

3. Verify your cgroup setup before trusting any limit. Under cgroup v2 with systemd delegation, rootless memory limits work; under cgroup v1, rootless Podman cannot create the controllers it needs and --memory is effectively ignored:

podman info --format '{{.Host.CgroupsVersion}}'
cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers

The controllers file must list memory. If it does not, the delegation is missing — add a systemd drop-in on the user slice:

# /etc/systemd/system/user@.service.d/delegate.conf
[Service]
Delegate=cpu cpuset io memory pids

Then sudo systemctl daemon-reload and log the user session out and back in.

4. Set an honest limit, and set swap along with it. --memory-swap is the combined memory+swap ceiling, not a separate swap allowance, so --memory=512m --memory-swap=512m means “512 MiB total, no swap”:

podman run -d --name api \
  --memory=1g --memory-swap=1g \
  docker.io/library/openjdk:21-jdk java -jar /app.jar

5. Size the runtime heap inside the limit rather than letting it discover host RAM. Modern JVMs are container-aware and will read the cgroup limit, but only if you let them use a percentage rather than a fixed host-derived default; Node.js needs an explicit --max-old-space-size:

# JVM: use ~75% of the container limit for the heap
podman run -d --memory=1g -e JAVA_TOOL_OPTIONS="-XX:MaxRAMPercentage=75.0" myapp:latest

# Node.js: cap old-space below the container limit (value is in MiB)
podman run -d --memory=1g -e NODE_OPTIONS="--max-old-space-size=768" mynode:latest

Leave headroom: the heap is not the process’s whole footprint. Metaspace, thread stacks, JIT code cache, and native allocations all live in the same cgroup.

6. Confirm the fix by watching the limit counters rather than waiting for another crash. memory.events incrementing max means you are still hitting the ceiling even if nothing died yet:

podman stats --no-stream api
CID=$(podman inspect --format '{{.Id}}' api)
grep -E '^(max|oom|oom_kill) ' /sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/user.slice/libpod-$CID.scope/memory.events

If oom_kill stays at zero over a full load cycle, the limit is now correct.

Prevention

• Always set --memory explicitly so a leaking container cannot take the host down with it.
• Run on cgroup v2 with memory delegated to the user slice, so rootless limits are actually enforced instead of silently ignored.
• Size JVM/Node/Python heaps as a percentage of the container limit, never as a fixed number copied from a bigger host.
• Handle SIGTERM in your application and use exec-form entrypoints so podman stop never needs to escalate.
• Set --stop-timeout to something longer than your slowest clean shutdown, and match it in any systemd unit.
• Alert on memory.events max and oom_kill counters, which rise before a container actually dies.

Related Errors

• Exited (143) — 128 + SIGTERM: a clean-ish termination signal, meaning the process was asked to stop and did, or died before escalation.
• Exited (125) — Podman itself failed to run the container, usually a bad flag or image reference rather than a runtime kill.
• Error: crun: setrlimit RLIMIT_NOFILE: Operation not permitted — a different resource-limit failure at startup; see Podman error: crun setrlimit RLIMIT_NOFILE.
• Error: OCI runtime error: container process exited — the runtime could not even start the process, so no exit code from your workload exists yet.

Frequently Asked Questions

Does exit code 137 always mean out of memory? No. It means SIGKILL, and the OOM killer is only one possible sender. Check podman inspect --format '{{.State.OOMKilled}}' first — if that is false and the kernel log has no oom-kill line, the signal came from userspace, most often podman stop escalating after its timeout.

Why is my --memory limit being ignored under rootless Podman? Because you are on cgroup v1, where an unprivileged user cannot create the memory controller for its own containers. Podman accepts the flag but the kernel never enforces it. Confirm with podman info --format '{{.Host.CgroupsVersion}}'; the fix is cgroup v2 plus Delegate=memory on the user slice.

My container has a 1 GiB limit and the JVM heap is 512 MiB — why did it still OOM? The heap is only part of the resident footprint. Metaspace, thread stacks, direct byte buffers, the JIT code cache, and glibc’s allocator arenas all count against the same cgroup. Either raise the limit or lower MaxRAMPercentage so total RSS stays under memory.max.

Can I stop the kernel from killing the container and get an error instead? Not usefully — the OOM killer exists precisely because there is no memory left to report an error with. The right move is to give the workload a limit it can live inside and watch memory.events for pressure. For more container runtime fixes, see the Podman guides.