🚨 Prometheus Error Guide: 'cannot allocate memory' on mmap — Fix vm.max_map_count Exhaustion

Overview

Prometheus memory-maps its TSDB head chunk files. When the process reaches the kernel’s per-process limit on memory-map areas (vm.max_map_count), the next mmap fails with ENOSPC/ENOMEM and Prometheus logs a chunk-mapping failure, usually during startup WAL replay or head compaction:

level=error component=tsdb msg="Failed to load chunks" err="mmap files, path:
  /prometheus/chunks_head: cannot allocate memory"

You may also see it as the process failing to open storage entirely:

level=error msg="Opening storage failed" err="mmap: cannot allocate memory"

The wording says “cannot allocate memory”, but this is not an OOM in the usual sense — the box can have gigabytes of free RAM. The kernel is refusing to create another memory-map area because the process has hit vm.max_map_count.

Symptoms

• Prometheus fails to start (crash-loops) with mmap ... cannot allocate memory during TSDB open / WAL replay.
• free -m shows plenty of available memory, yet the allocation “fails”.
• The failure correlates with a large number of head chunk files (high active series, high churn).
• On Kubernetes the pod is not OOMKilled (no exit code 137, no Killed event) — it exits non-zero from the TSDB error.
• /proc/<pid>/maps line count is near the vm.max_map_count value.

Common Root Causes

• vm.max_map_count too low — the default (often 65530) is exceeded by a Prometheus instance with high active-series counts and many memory-mapped head chunks.
• Series churn — frequent target rollouts create huge numbers of short-lived series, multiplying head chunk files and map areas.
• Large head before compaction — a long compaction interval or delayed compaction leaves many head chunks mapped at once.
• Multiple mmap-heavy processes on the same host competing for kernel map areas (though the limit is per-process, misconfigured sysctls affect all).
• Container/host mismatch — the sysctl set on the node is not applied inside the container’s context, or an init system reset it.
• Downstream mmap-heavy tools (Thanos sidecar reading the same directory) adding to map pressure.

Diagnostic Workflow

First confirm the current kernel limit and how close the process is to it. If the map count is near the limit, that is your answer:

# Current per-process map-area limit
cat /proc/sys/vm/max_map_count

# How many map areas the Prometheus process is actually using
pid=$(pgrep -o prometheus)
wc -l < /proc/$pid/maps

Rule out a real OOM before touching the sysctl — check for OOMKills and actual memory pressure:

# No OOMKill => this is the map-count limit, not RAM
dmesg -T | grep -i 'killed process' | tail
free -m

On Kubernetes, confirm the exit reason is the TSDB error and not exit code 137:

kubectl describe pod prometheus-0 | sed -n '/Last State/,/Ready/p'
kubectl logs prometheus-0 --previous | grep -i 'mmap\|cannot allocate memory'

Gauge the workload driving map count — high active series and churn:

prometheus_tsdb_head_series
rate(prometheus_tsdb_head_series_created_total[5m])
prometheus_tsdb_mmap_chunk_corruptions_total

Example Root Cause Analysis

A prometheus-0 StatefulSet pod entered CrashLoopBackOff after a cluster grew from 400k to 1.1M active series. Logs showed:

level=error component=tsdb msg="Failed to load chunks"
  err="mmap files, path: /prometheus/chunks_head: cannot allocate memory"

The pod had no OOMKill event and its memory limit was not reached, so the on-call first ruled out RAM. Checking the node:

$ cat /proc/sys/vm/max_map_count
65530
$ wc -l < /proc/$(pgrep -o prometheus)/maps
65529

The process was one map area under the default limit. The series growth had produced enough head chunk files that Prometheus could not mmap another during WAL replay. Raising vm.max_map_count on the node to 262144 and restarting let the TSDB open cleanly; the map count settled around 90k. To keep it from recurring, the team also cut series churn by dropping a high-cardinality pod_template_hash label via metric_relabel_configs.

Prevention Best Practices

• Raise vm.max_map_count well above the default on any Prometheus host — 262144 is a common, safe value for large instances. Persist it in /etc/sysctl.d/:

  echo 'vm.max_map_count=262144' | sudo tee /etc/sysctl.d/99-prometheus.conf
  sudo sysctl --system

• On Kubernetes, set the node sysctl via a DaemonSet/initContainer (privileged) or node bootstrap, since pod securityContext.sysctls does not cover vm.max_map_count (it is not namespaced).
• Control cardinality and churn with metric_relabel_configs to drop volatile labels (hashes, ephemeral IDs) that create short-lived series.
• Alert on prometheus_tsdb_head_series growth and on the process map count trending toward the limit before it fails.
• Keep head compaction healthy; investigate delayed compaction that leaves excess head chunks mapped.
• Size the instance’s active series against the configured max_map_count, and shard/scale out before a single server outgrows the limit.

Quick Command Reference

# Confirm the limit vs. actual usage
cat /proc/sys/vm/max_map_count
wc -l < /proc/$(pgrep -o prometheus)/maps

# Rule out a real OOM
dmesg -T | grep -i 'killed process' | tail
free -m

# Raise and persist the limit
echo 'vm.max_map_count=262144' | sudo tee /etc/sysctl.d/99-prometheus.conf
sudo sysctl --system

# Kubernetes: confirm the crash reason
kubectl logs prometheus-0 --previous | grep -i 'cannot allocate memory'

Conclusion

A Prometheus mmap ... cannot allocate memory error is the kernel’s vm.max_map_count ceiling, not a shortage of RAM — which is why the box shows free memory and the pod is never OOMKilled. Confirm by comparing /proc/<pid>/maps line count against /proc/sys/vm/max_map_count, raise the limit to 262144 (persisted via sysctl or a node DaemonSet), and attack the underlying driver by cutting series churn with relabeling. Treat rising active-series and map counts as a scale signal to shard before the next instance hits the same wall.