🚨 Cut your Docker MTTR — resolve production incidents fast

Severity guidance: P1 if an OOM-kill takes down a customer-facing service or the container flaps (kill → restart → kill). P2 if a single non-critical container is killed but the service stays up (e.g. one replica of several). P3 in dev/CI, or a batch job that can simply be re-run with more memory. Time to resolution estimate: 15–45 min to mitigate (raise the limit / restart with correct heap flags); hours to days to root-cause a genuine memory leak. Blast radius: Usually a single container. Escalates to the whole host when there is no per-container memory limit and the kernel OOM-killer starts reaping arbitrary processes (including dockerd or sshd) to save the host.

Symptoms

• docker ps -a shows the container Exited (137).
• docker inspect reports "OOMKilled": true.
• Application logs stop mid-write with no shutdown/cleanup lines — the process was SIGKILLed, so it never ran its exit handlers.
• Kernel log (dmesg/journal) shows one of:
  • Memory cgroup out of memory: Killed process 12345 (java) ... → the container hit its own cgroup limit.
  • Out of memory: Killed process 12345 (node) ... (no "Memory cgroup" prefix) → the host ran out of memory globally.
• Orchestrator equivalents: Kubernetes OOMKilled, Nomad OOM, ECS OutOfMemoryError.
• Metric pattern: container_memory_working_set_bytes climbs to the limit, then the series resets to near-zero as the container restarts (a sawtooth).

Note: exit code 137 = 128 + 9 (SIGKILL). A manual docker kill also produces 137. The OOMKilled flag is what distinguishes an OOM from an operator kill — always check it (triage step 2).

Immediate triage (first 5 minutes)

1. Confirm it was an OOM, not a normal kill.

docker inspect --format \
  'exit={{.State.ExitCode}} oom={{.State.OOMKilled}} restarts={{.RestartCount}} status={{.State.Status}}' \
  <container>

Broken (OOM): exit=137 oom=true restarts=6 status=restarting Healthy: exit=0 oom=false restarts=0 status=running

2. Determine whether it was the container's limit or the whole host.

# Container's own cgroup was exhausted:
sudo dmesg -T | grep -iE 'killed process|out of memory' | tail -5

Container-limit kill: Memory cgroup out of memory: Killed process 12345 (java) total-vm:5242880kB, anon-rss:2087600kB Host-wide kill (no "cgroup"): Out of memory: Killed process 12345 (node) total-vm:...

3. See the configured limit and current usage.

docker stats --no-stream <container>

CONTAINER   MEM USAGE / LIMIT     MEM %   ...
api         1.998GiB / 2GiB       99.9%   ...

A MEM % pinned near 100 right before the kill confirms the container was starved at its limit. If LIMIT shows the host total (e.g. 15.6GiB) the container has no memory limit set — jump to Diagnosis cause C.

Diagnosis

Decision tree:

• If dmesg says "Memory cgroup out of memory" → the container hit its own limit → Cause A (limit too low or leak).
• If dmesg says plain "Out of memory" with no cgroup line → host-wide pressure → Cause B (host oversubscribed / no per-container limits).
• If docker stats LIMIT == host RAM (no --memory set) → Cause C (unbounded container).
• If the process is a JVM/Node/Python runtime and RSS ≈ limit → Cause D (runtime heap not aware of the cgroup limit).

Read the effective limit straight from the cgroup (authoritative — survives daemon-config drift):

# Which cgroup version is this host on?
stat -fc %T /sys/fs/cgroup     # cgroup2fs = v2 (unified) ; tmpfs = v1

# cgroup v2 (Docker 20.10+ on modern distros — Ubuntu 22.04+, Fedora, RHEL 9):
cat /sys/fs/cgroup/system.slice/docker-$(docker inspect -f '{{.Id}}' <container>).scope/memory.max
cat /sys/fs/cgroup/system.slice/docker-$(docker inspect -f '{{.Id}}' <container>).scope/memory.current
# memory.events shows the cumulative kill count:
grep oom_kill /sys/fs/cgroup/system.slice/docker-$(docker inspect -f '{{.Id}}' <container>).scope/memory.events
#   oom_kill 6

# cgroup v1 (older hosts — Ubuntu 20.04 default, Amazon Linux 2):
CID=$(docker inspect -f '{{.Id}}' <container>)
cat /sys/fs/cgroup/memory/docker/$CID/memory.limit_in_bytes
cat /sys/fs/cgroup/memory/docker/$CID/memory.usage_in_bytes
cat /sys/fs/cgroup/memory/docker/$CID/memory.stat   # rss vs cache breakdown

cgroup v1 gotcha: memory.usage_in_bytes includes page cache, so a container doing heavy file I/O can look "full" without a real leak. Look at rss in memory.stat, not total usage. cgroup v2's memory.current with memory.stat anon is the cleaner signal. This is a common false-positive when triaging on v1 hosts.

Cause D check — is the runtime heap-aware?

# JVM: is container support on and what's the max heap it computed?
docker exec <container> java -XX:+PrintFlagsFinal -version 2>/dev/null \
  | grep -E 'UseContainerSupport|MaxHeapSize|MaxRAMPercentage'
# Node: what did V8 pick for the old-space?
docker exec <container> node -e 'console.log((require("v8").getHeapStatistics().heap_size_limit/1048576).toFixed(0)+" MiB")'

• JVM: UseContainerSupport is on by default since 8u191 / JDK 10+. On older JVMs it reads the host RAM and sizes the heap far above the container limit → guaranteed OOM. Fix in Cause D.
• Node: V8's default old-space cap is ~2048 MiB on 64-bit regardless of the container limit. If the container limit is <2 GiB and the app is memory-hungry, V8 will happily grow past the cgroup limit and get killed before it ever hits its own GC ceiling.

Resolution

Cause A — limit too low for legitimate working set

Raise the limit on the running container (no recreate needed) to stop the bleeding, then bake it into the manifest:

docker update --memory 4g --memory-swap 4g <container>   # --memory-swap = --memory disables swap
# verify the new ceiling:
docker inspect -f '{{.HostConfig.Memory}}' <container>    # 4294967296
docker stats --no-stream <container>                      # MEM % should now sit well below 100

docker update changes the cgroup live but does not persist to your compose file — update mem_limit/deploy.resources.limits.memory there too, or the next up/recreate reverts it.

Compose (v2 spec):

services:
  api:
    mem_limit: 4g          # non-swarm
    # or, for swarm/deploy:
    deploy:
      resources:
        limits:   { memory: 4g }
        reservations: { memory: 2g }

Cause B — host oversubscribed

Sum the limits vs host RAM and find who has none:

docker ps -q | xargs docker inspect \
  -f '{{.Name}} limit={{.HostConfig.Memory}}' | sed 's/limit=0/limit=UNBOUNDED/'
free -m

Set limits on the unbounded offenders (Cause C), or move a container to another host. Verify no more host-level kills: dmesg -T | grep -i 'out of memory' | tail.

Cause C — no memory limit at all

Set one. Pick the limit from observed steady-state RSS + ~30% headroom:

docker update --memory 1g --memory-swap 1g <container>

Then persist it in compose and add a reservations floor so the scheduler accounts for it.

Cause D — runtime not cgroup-aware

• Modern JVM (recommended): let it size from the cgroup.

  # In the container env / entrypoint:
  JAVA_TOOL_OPTIONS="-XX:MaxRAMPercentage=75.0 -XX:InitialRAMPercentage=50.0"
  

  Verify: docker exec <c> java -XX:+PrintFlagsFinal -version | grep MaxHeapSize should now be ~75% of the container limit.
• Legacy JVM (<8u191): pin it explicitly — -Xmx1536m for a 2 GiB container (leave headroom for metaspace, threads, and direct buffers; the heap is not the whole RSS).
• Node: set the old-space cap below the container limit, leaving ~20–25% for non-heap:

  NODE_OPTIONS="--max-old-space-size=1536"   # for a 2 GiB container
  

  Verify: the heap_size_limit from the triage command now reflects your value.

After any fix, confirm stability:

docker events --filter container=<container> --filter event=oom &   # should print NOTHING going forward
watch -n5 'docker stats --no-stream <container>'                     # MEM % steady, not ramping

If that didn't work

• Genuine leak (usage ramps monotonically over hours regardless of limit): capture a heap profile before the next kill. JVM: jcmd 1 GC.heap_dump /tmp/heap.hprof (or -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp). Node: run with --heapsnapshot-near-heap-limit=3 or kill -USR2 with heapdump. Analyze offline; do not debug in the P1 window.
• Page-cache false positive (cgroup v1 only): high memory.usage_in_bytes but low rss → not a real leak; the kernel will reclaim cache under pressure. Don't raise the limit for this; if kills still happen, it's dirty page writeback stalling — see RB-23 (log/stdout backpressure) and check memory.stat dirty/writeback.
• --oom-kill-disable set: the container won't be killed but will freeze at the limit instead (worse). Remove it unless you truly want to pause-and-page.
• Kernel picking the wrong victim: if dockerd/sshd is being killed instead of the hog, the hog has no limit (Cause C) — the kernel scores by absolute RSS. Fix by limiting the hog, not by tuning oom_score_adj.
• Init/forking apps: RSS counted across all child PIDs in the cgroup — a fork bomb or connection-per-process model can OOM a "small" app. See RB-16 (zombie/PID-1) for process accounting.

Prevention

• Always set mem_limit + reservations on every production container. Unbounded containers are the root cause of host-wide OOMs (Cause B/C).
• Make runtimes cgroup-aware (Cause D): MaxRAMPercentage for JVM, --max-old-space-size for Node, --memory awareness for Python (e.g. resource.setrlimit or a supervisor).
• Alert before the kill, not after. Example Prometheus rule (cAdvisor metrics):

  groups:
    - name: container-memory
      rules:
        - alert: ContainerMemoryNearLimit
          expr: |
            container_memory_working_set_bytes{container!=""}
              / (container_spec_memory_limit_bytes{container!=""} > 0)
            > 0.90
          for: 5m
          labels:    { severity: warning }
          annotations:
            summary: "{{ $labels.container }} above 90% of its memory limit"
            description: "Working set has been >90% of the cgroup limit for 5m — OOM-kill likely."
        - alert: ContainerOOMKilled
          expr: increase(container_oom_events_total[5m]) > 0
          labels:    { severity: critical }
          annotations:
            summary: "{{ $labels.container }} was OOM-killed"
  

• Dashboard container_memory_working_set_bytes / container_spec_memory_limit_bytes per container so leaks show as a rising ramp days before they page you.
• Related runbooks: RB-02 (the OOM→restart→OOM loop this creates), RB-18 (noisy-neighbor / who's eating the host), RB-23 (stdout backpressure that inflates memory).