🚨 Linux Error Guide: 'Out of memory: Killed process 1234 (name)' — Tame the OOM Killer

Overview

A service disappears without a crash log of its own, and the kernel ring buffer holds the explanation:

Out of memory: Killed process 1234 (name)

This is the Linux Out-Of-Memory (OOM) killer at work. When the kernel cannot satisfy a memory allocation and cannot reclaim enough by dropping caches or swapping, it invokes the OOM killer to free memory by terminating a process. The victim is not random: the kernel scores every process by memory footprint (weighted by oom_score_adj) and kills the one with the highest oom_score, aiming to recover the most memory while sparing critical system processes.

The message can originate from two places. A system-wide OOM means physical RAM plus swap is exhausted. A cgroup OOM means a single control group hit its memory.max limit — extremely common under systemd services and containers, where one workload is capped while the host still has free RAM. Reading the log carefully tells you which case you are in, and that determines the fix.

Symptoms

• A long-running daemon or container vanishes abruptly; systemctl status shows it was killed by signal SIGKILL (9) or reports oom-kill.
• dmesg or the journal contains Out of memory: Killed process <pid> (<name>) followed by a memory usage table listing the top consumers.
• The system briefly becomes unresponsive or laggy just before the kill, as the kernel thrashes trying to reclaim memory.
• Under systemd, systemctl status myservice includes a line like Main process exited, code=killed, status=9/KILL and Memory: 512.0M (max: 512.0M).
• Restarts appear healthy, then the same process is killed again minutes or hours later — the signature of a memory leak or persistent under-sizing.

Common Root Causes

1. Genuine memory exhaustion. The workload’s working set exceeds physical RAM, and there is little or no swap to absorb the overflow.
2. A leaking process. One application steadily grows its resident set over hours or days until it triggers the killer.
3. cgroup / systemd memory limit reached. A unit’s MemoryMax or a container’s memory.max is set below what the workload needs, so the cgroup OOMs even though host RAM is plentiful.
4. No swap configured. Without swap the kernel has no cushion for transient spikes, so bursts that would otherwise page out instead trigger an immediate kill.
5. Overcommit settings. With vm.overcommit_memory=2 (strict), allocations are refused earlier; with the default heuristic mode, the kernel may hand out more than it can back, deferring the reckoning to allocation time.

Diagnostic Workflow

First confirm an OOM kill happened and identify the victim. dmesg -T adds human-readable timestamps:

dmesg -T | grep -i 'killed process'
# [Fri Jul  3 09:14:22 2026] Out of memory: Killed process 1234 (myapp) total-vm:...

On systems that log the kernel to a file, or via the journal, use whichever is available:

grep -i oom /var/log/syslog          # Debian/Ubuntu
journalctl -k | grep -i oom          # any systemd host, kernel messages

The full OOM report includes a per-process table (RSS, oom_score_adj) — scan it to see which processes dominated memory at the moment of the kill. To understand why a specific process was chosen, inspect its live score while it runs:

cat /proc/<pid>/oom_score          # higher = more likely to be killed
cat /proc/<pid>/oom_score_adj      # bias: -1000 (never) to +1000 (first)

Check the overall memory picture and whether swap exists:

free -h
# note the Swap line: 0B total means no cushion

Review the overcommit policy, which governs how generously the kernel grants allocations:

sysctl vm.overcommit_memory vm.overcommit_ratio
# vm.overcommit_memory = 0   (0 heuristic, 1 always, 2 never/strict)
# vm.overcommit_ratio = 50

If the victim runs under systemd or a container, the kill was very likely a cgroup limit. Check the unit’s configured cap and its cgroup’s OOM accounting:

systemctl show myservice -p MemoryMax -p MemoryCurrent
# read the cgroup's own counter (v2):
cat /sys/fs/cgroup/system.slice/myservice.service/memory.events
# oom_kill 3        <-- this cgroup has been OOM-killed three times
cat /sys/fs/cgroup/system.slice/myservice.service/memory.max

A nonzero oom_kill in memory.events while host free -h shows ample memory proves this is a limit problem, not a host exhaustion problem — the fix is to raise the limit or reduce the workload, not to add RAM.

Example Root Cause Analysis

An API service is killed every few hours. journalctl -k | grep -i oom shows Out of memory: Killed process 1234 (api) each time, but the host has 32 GB of RAM and free -h never dips below 20 GB free. That contradiction points away from host exhaustion.

Inspecting the unit, systemctl show api -p MemoryMax returns MemoryMax=1073741824 (1 GB), and cat /sys/fs/cgroup/system.slice/api.service/memory.events shows oom_kill 7. The service’s own cgroup is the boundary being hit, not the machine. Watching MemoryCurrent over time reveals it climbs steadily from 300 MB to 1 GB and then dies — a leak, but one made fatal by a limit that is too tight for even normal peak load.

The team applies two changes. Short term, they raise the cap with a drop-in (MemoryMax=2G) via systemctl edit api to stop the immediate outages. Longer term, they add memory profiling to find the leak and set MemoryHigh=1500M so the cgroup throttles and reclaims before hitting the hard MemoryMax, turning an abrupt kill into graceful back-pressure.

Prevention Best Practices

• Right-size memory based on observed peak usage plus headroom, rather than guessing; use MemoryCurrent trends to set realistic limits.
• Add swap (even a few GB, or zram) so transient spikes page out instead of triggering an immediate kill.
• Set MemoryMax and MemoryHigh on systemd units so one runaway service cannot starve the whole host, and so it throttles before it dies.
• Protect critical processes with a negative oom_score_adj (for example via OOMScoreAdjust=-500 on a systemd unit) so the killer targets expendable workloads first.
• Fix leaks at the source — persistent OOMs usually indicate a leak; profiling beats endlessly raising limits.
• Alert on memory.events oom_kill so repeated cgroup kills surface before users notice outages.

Quick Command Reference

dmesg -T | grep -i 'killed process'          # recent OOM kills with timestamps
grep -i oom /var/log/syslog                  # Debian/Ubuntu log search
journalctl -k | grep -i oom                  # kernel OOM messages (systemd)
cat /proc/<pid>/oom_score                    # current kill likelihood
cat /proc/<pid>/oom_score_adj                # kill bias (-1000..+1000)
free -h                                       # RAM and swap at a glance
sysctl vm.overcommit_memory vm.overcommit_ratio   # overcommit policy
systemctl show myservice -p MemoryMax -p MemoryCurrent   # unit limit and usage
cat /sys/fs/cgroup/system.slice/myservice.service/memory.events   # oom_kill count
cat /sys/fs/cgroup/system.slice/myservice.service/memory.max      # cgroup hard limit

Conclusion

Out of memory: Killed process is the kernel making a hard choice: free memory now, or let the whole system stall. Your first job is to determine whether the boundary was the host (RAM plus swap exhausted) or a cgroup limit — free -h versus memory.events settles it quickly. From there, right-size limits, add swap, protect critical processes with oom_score_adj, and chase down leaks rather than repeatedly nudging caps upward. Handled well, the OOM killer becomes a safety net you rarely trip rather than a recurring outage.