🚨 VictoriaMetrics Error Guide: 'Out of memory: Killed process 1234 (victoria-metrics)' — Cap Memory and Concurrency

Overview

The Linux kernel OOM killer terminates the VictoriaMetrics process when it grows past what the host can back with physical memory. You see it in dmesg or journalctl right before the service restarts:

Out of memory: Killed process 1234 (victoria-metrics) total-vm:38452120kB, anon-rss:31090112kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:62000kB oom_score_adj:0

On a cluster deployment the same message names vmselect or vmstorage instead of victoria-metrics. The process disappears, systemd restarts it, in-flight queries fail, and — most damaging — vmstorage may replay its write-ahead log on startup, adding minutes of unavailability. This is almost never a leak; it is VictoriaMetrics correctly using the RAM it was allowed, then the working set of a heavy query or high-cardinality ingestion pushing total usage past the physical limit.

Symptoms

• The victoria-metrics, vmselect, or vmstorage process restarts unexpectedly with no panic in its own log — the kill happens outside the process.
• journalctl -u victoriametrics shows main: shutting down was never logged; the service just died and came back.
• dmesg -T | grep -i oom shows Out of memory: Killed process ... (victoria-metrics) or (vmselect).
• Grafana panels go blank or return 502/503 during the restart window.
• Memory climbs steeply right after a specific dashboard load or a wide-range MetricsQL query, then the process dies.
• process_resident_memory_bytes in the self-scraped metrics rises toward the cgroup/host limit before each kill.

Common Root Causes

• Heavy queries with huge time series fan-out — a MetricsQL query touching millions of series (for example an unbounded sum(rate(...)) over a high-cardinality label) buffers points in RAM on vmselect.
• Too many concurrent queries — dashboards auto-refreshing across many users saturate -search.maxConcurrentRequests, each holding a working set.
• High ingestion cardinality — a churny label (pod name, request ID, full URL) explodes active time series on vmstorage, inflating the in-memory index.
• -memory.allowedPercent set too high for a shared host, so VictoriaMetrics sizes its caches for more RAM than actually exists.
• No cgroup/memory limit on a container, letting VictoriaMetrics assume the whole node’s RAM while other processes compete.
• Undersized nodes relative to active series and query load — the honest case where you simply need more RAM or more vmstorage shards.

Diagnostic Workflow

Confirm it was an OOM kill and identify which binary died:

sudo dmesg -T | grep -i 'out of memory'
sudo journalctl -u victoriametrics --since '30 min ago' | grep -iE 'killed|oom|restart'
sudo journalctl -u vmselect -u vmstorage --since '30 min ago'

Scrape the process’s own metrics and inspect memory and query load. On single-node use port 8428; on cluster use vmselect 8481 and vmstorage 8482:

curl -s http://<node>:8428/metrics | grep -E 'process_resident_memory_bytes|vm_available_memory_bytes|vm_cache_size_bytes'
curl -s http://<node>:8481/metrics | grep -E 'vm_concurrent_select|vm_concurrent_insert'

Look for the cardinality and query-cost signals:

# Active time series and per-metric row counts on vmstorage
curl -s http://<node>:8482/metrics | grep -E 'vm_rows|vm_new_timeseries_created_total'

Use the cardinality explorer built into vmui to find the offending metric or label. Open vmui and run the Cardinality tab, or query TSDB status directly:

# Single-node vmui: http://<node>:8428/vmui  (Cardinality tab)
curl -s 'http://<node>:8428/api/v1/status/tsdb' | jq '.data.seriesCountByMetricName[0:10]'

Reproduce the memory spike by running the suspect MetricsQL in vmui and watching process_resident_memory_bytes climb. A query like the following, over a churny label, is a classic offender:

sum(rate(http_requests_total[5m])) by (request_id)

Check the effective memory sizing flag the process started with:

sudo journalctl -u victoriametrics | grep -i 'allowedPercent\|available memory'
ps -o pid,rss,cmd -C victoria-metrics

Example Root Cause Analysis

A team ran single-node VictoriaMetrics on a 32 GB host with -memory.allowedPercent=80. Every weekday at 09:00 the process was OOM-killed. dmesg showed Out of memory: Killed process 1234 (victoria-metrics) with anon-rss near 31 GB.

curl http://vm01:8428/api/v1/status/tsdb revealed that http_requests_total had 4.1 million series — because an instrumentation change had added a request_id label, giving every request a unique series. When the morning traffic dashboard loaded a sum(rate(...)) by (request_id) panel, vmselect had to fetch and buffer all of those series at once. Combined with -memory.allowedPercent=80 sizing caches for 25.6 GB, total usage blew past 32 GB and the kernel intervened.

The fix was two-pronged. Immediately, they dropped the offending label at ingestion using vmagent relabeling (action: labeldrop on request_id), which collapsed active series from 4.1M to ~9K. Then they lowered -memory.allowedPercent=60 to leave headroom and set -search.maxConcurrentRequests=8 so a burst of dashboard refreshes could not each grab a large working set. After the change, process_resident_memory_bytes peaked at 14 GB during the morning load and the OOM kills stopped entirely.

Prevention Best Practices

• Set -memory.allowedPercent (default 60) conservatively, and never above what leaves room for the OS page cache and other processes on a shared host.
• Cap query concurrency with -search.maxConcurrentRequests and bound cost with -search.maxPointsPerTimeseries and -search.maxQueryDuration so one expensive query cannot exhaust RAM.
• Control cardinality at the source: drop high-churn labels (request_id, pod hashes, full URLs) with vmagent relabeling before they reach vmstorage.
• Run the vmui Cardinality explorer weekly and alert on vm_new_timeseries_created_total spikes to catch label explosions early.
• Put a hard memory limit (cgroup / container --memory / systemd MemoryMax) on the process so a bad query degrades gracefully instead of taking the node down.
• Scale out: in cluster mode add vmstorage nodes to spread active series, and add vmselect replicas to spread query load, rather than pushing a single node past its RAM.
• Alert on process_resident_memory_bytes / vm_available_memory_bytes > 0.85 to intervene before the kernel does.

Quick Command Reference

# Confirm the OOM kill
sudo dmesg -T | grep -i 'out of memory'
sudo journalctl -u victoriametrics --since '30 min ago' | grep -iE 'killed|oom'

# Memory + query load (single-node 8428 / cluster vmselect 8481, vmstorage 8482)
curl -s http://<node>:8428/metrics | grep -E 'process_resident_memory_bytes|vm_available_memory_bytes'
curl -s http://<node>:8481/metrics | grep vm_concurrent_select

# Find cardinality offenders
curl -s 'http://<node>:8428/api/v1/status/tsdb' | jq '.data.seriesCountByMetricName[0:10]'
# or open http://<node>:8428/vmui  -> Cardinality tab

# Recommended guardrail flags
# -memory.allowedPercent=60
# -search.maxConcurrentRequests=8
# -search.maxPointsPerTimeseries=30000
# -search.maxQueryDuration=60s

Conclusion

An Out of memory: Killed process 1234 (victoria-metrics) line is the kernel telling you the working set outgrew physical RAM — usually driven by a high-cardinality metric feeding an expensive MetricsQL query, amplified by a generous -memory.allowedPercent. Confirm the kill in dmesg/journalctl, use the vmui Cardinality explorer and /api/v1/status/tsdb to find the label that exploded, then attack both ends: cut cardinality at ingestion with vmagent relabeling and cap query concurrency and cost with -search.* flags. Leave real headroom with -memory.allowedPercent, put a memory limit on the process, and scale vmstorage/vmselect horizontally when the load is genuinely large. Do that and the OOM killer stays quiet.