🚨 Linux Error Guide: 'fork: Cannot allocate memory' — Fix ENOMEM from Limits and Overcommit

Overview

When fork() or mmap() returns ENOMEM, the shell or application reports one of two messages:

-bash: fork: Cannot allocate memory

or, from a process calling mmap() directly (common with JVMs, Elasticsearch, and memory-mapped databases):

mmap failed: Cannot allocate memory

The critical insight is that this error is frequently not a RAM exhaustion problem. The kernel can refuse fork() or mmap() for several reasons that have nothing to do with how much physical memory is available: the calling process or user has hit a process count limit, the system-wide thread ceiling has been reached, the number of virtual memory mappings exceeds a kernel tunable, the virtual address space limit is too low, or the overcommit accounting policy rejects the request. Seeing free -h report gigabytes of available RAM while bash refuses to fork is a classic sign that a kernel limit — not RAM — is the real constraint.

Symptoms

• Every attempt to run a command from the shell fails with the fork error; the shell itself continues to exist but cannot spawn children.
• SSH connections fail silently or disconnect immediately after authentication because the sshd cannot fork a child.
• Existing processes continue to run normally — only the creation of new processes or threads is blocked.
• Java or Elasticsearch logs mmap failed: Cannot allocate memory at startup or when allocating direct byte buffers.
• free -h may show gigabytes of free and available RAM, confirming this is a limit rather than a capacity issue.
• dmesg shows no OOM events.

Common Root Causes

• Per-user nproc limit (ulimit -u / RLIMIT_NPROC) — the shell’s user has reached the maximum number of simultaneously running processes. The default on many distributions is 4096 or lower for non-root users.
• System-wide PID ceiling (kernel.pid_max) — the entire system has reached the maximum number of process IDs. Default is 32768 on 32-bit; 4194304 on 64-bit kernels.
• kernel.threads-max — a separate ceiling on the total number of kernel threads system-wide.
• cgroup pids.max — a container or systemd service unit has a TasksMax= limit, and the cgroup has exhausted its PID budget.
• vm.max_map_count too low — each call to mmap() consumes one VMA (Virtual Memory Area) entry. The default of 65530 is insufficient for Elasticsearch and JVMs with large heaps, which can require hundreds of thousands of mappings.
• Overcommit policy too strict (vm.overcommit_memory=2) — in strict overcommit mode the kernel refuses allocations that would exceed (RAM × overcommit_ratio / 100) + swap. A low overcommit_ratio (default 50) can block allocation even when RAM appears available.
• Virtual address space ulimit (ulimit -v) — if an explicit address-space ceiling is set for the user or process, large mmap() calls fail even with free RAM.
• Genuine RAM and swap exhaustion — the system has no physical RAM or swap remaining, so any new allocation genuinely cannot succeed.

Diagnostic Workflow

Print all resource limits for the current shell session:

ulimit -a

Focus on max user processes (nproc) and virtual memory. If nproc shows a low value like 4096, compare it to the current process count for this user.

Count how many processes the user is currently running:

ps -u "$USER" --no-headers | wc -l

Check the system-wide PID and thread ceilings:

cat /proc/sys/kernel/pid_max
cat /proc/sys/kernel/threads-max

Count all threads currently alive on the system:

ps -eLf | wc -l

If the thread count is close to threads-max, that is the bottleneck.

Inspect the overcommit policy:

cat /proc/sys/vm/overcommit_memory
cat /proc/sys/vm/overcommit_ratio

A value of 2 for overcommit_memory enables strict accounting. In that mode, compute the commit limit:

grep -E 'CommitLimit|Committed_AS' /proc/meminfo

If Committed_AS is approaching CommitLimit, new allocations will be refused.

Check the current vm.max_map_count and how many VMAs a suspect process already holds:

sysctl vm.max_map_count
# Count VMA entries for a specific process (e.g., PID 1842)
wc -l /proc/1842/maps

If the per-process map count is near 65530 (the default max_map_count), mmap() calls will fail.

Check actual RAM and swap:

free -h

If available is non-trivial (hundreds of megabytes or more) but the fork error persists, RAM is not the issue.

Scan dmesg for OOM events to confirm genuine exhaustion is not involved:

dmesg | grep -iE 'oom|out of memory|cannot allocate'

For containerized workloads, check cgroup PID limits:

# cgroup v2 — find the relevant cgroup for a service
systemctl status <service-name>
cat /sys/fs/cgroup/system.slice/<service-name>.service/pids.current
cat /sys/fs/cgroup/system.slice/<service-name>.service/pids.max

If pids.current equals pids.max, the cgroup has hit its TasksMax ceiling.

Example Root Cause Analysis

An Elasticsearch 8.x node fails immediately on startup with the following in the Elasticsearch log:

ERROR: [1] bootstrap checks failed. You must address the point(s) below and restart.
bootstrap check failure [1] of [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

From the OS side, attempting to start the process manually produces:

mmap failed: Cannot allocate memory

Verify the current kernel setting:

sysctl vm.max_map_count

vm.max_map_count = 65530

Apply the fix immediately without a reboot:

sudo sysctl -w vm.max_map_count=262144

Verify:

sysctl vm.max_map_count

vm.max_map_count = 262144

Persist the change across reboots:

echo 'vm.max_map_count=262144' | sudo tee /etc/sysctl.d/99-elasticsearch.conf
sudo sysctl -p /etc/sysctl.d/99-elasticsearch.conf

Start Elasticsearch again — the mmap error disappears and the bootstrap check passes.

Prevention Best Practices

• Raise vm.max_map_count on Elasticsearch and JVM hosts at provisioning time. Add it to your configuration management templates so new nodes get it automatically.
• Tune nproc in /etc/security/limits.conf for accounts that run high-thread services. The format is:

# /etc/security/limits.conf
appuser  soft  nproc  65536
appuser  hard  nproc  65536

This applies to PAM-authenticated logins. For systemd services, set TasksMax= in the unit file or override with systemctl edit.

• Choose an appropriate overcommit policy for your workload. Most servers should use vm.overcommit_memory=1 (always allow, rely on the OOM killer as backstop) rather than the strict mode. Only use vm.overcommit_memory=2 when you have a specific reason and have validated overcommit_ratio is set high enough.
• Monitor process and thread counts with Prometheus node_exporter metrics (node_processes_threads, node_processes_pids). Alert when either metric exceeds 70% of the corresponding kernel ceiling.
• Set TasksMax in systemd units for services that have a bounded expected thread count so a runaway thread leak is caught before it affects the whole system.

Quick Command Reference

# Current shell resource limits
ulimit -a

# Process count for current user
ps -u "$USER" --no-headers | wc -l

# System-wide PID and thread limits
cat /proc/sys/kernel/pid_max
cat /proc/sys/kernel/threads-max

# Total threads currently running
ps -eLf | wc -l

# Overcommit policy and ratio
cat /proc/sys/vm/overcommit_memory
cat /proc/sys/vm/overcommit_ratio

# Commit accounting
grep -E 'CommitLimit|Committed_AS' /proc/meminfo

# VMA mapping limit and per-process map count
sysctl vm.max_map_count
wc -l /proc/<pid>/maps

# Apply max_map_count fix immediately
sudo sysctl -w vm.max_map_count=262144

# Persist max_map_count across reboots
echo 'vm.max_map_count=262144' | sudo tee /etc/sysctl.d/99-max-map-count.conf

# Raise nproc limit (add to /etc/security/limits.conf)
# appuser  soft  nproc  65536
# appuser  hard  nproc  65536

# Check cgroup PID usage (cgroup v2)
cat /sys/fs/cgroup/system.slice/<service>.service/pids.current
cat /sys/fs/cgroup/system.slice/<service>.service/pids.max

# Check actual RAM (to rule out genuine exhaustion)
free -h

# Scan dmesg for OOM (genuine exhaustion evidence)
dmesg | grep -iE 'oom|out of memory'

Conclusion

fork: Cannot allocate memory and mmap failed: Cannot allocate memory are deceptive errors because they sound like hardware resource problems when they are usually software limit problems. Before assuming you need more RAM, check ulimit -u against your user’s current process count, verify that vm.max_map_count is high enough for memory-mapped applications like Elasticsearch, inspect pids.max in the relevant cgroup, and review the overcommit policy. In most production incidents involving this error, adjusting a single kernel parameter or a limits.conf entry resolves it without any hardware change.