🚨 VictoriaMetrics Error: 'the query requires more than -search.maxMemoryPerQuery bytes of memory' — Cause, Fix, and Troubleshooting Guide

Overview

vmselect (or single-node victoria-metrics) estimates how much memory a query will need before executing it, and rejects the request if that estimate exceeds the per-query budget:

cannot execute query: the query requires more than -search.maxMemoryPerQuery=0 bytes of memory; reduce the time range or add more specific label filters

-search.maxMemoryPerQuery bounds how much RAM any single query may use so one heavy request cannot exhaust the process and OOM the node. The estimate scales with the number of series and datapoints the query aggregates over, and the effective budget is shared and constrained by total RAM and -memory.allowedPercent. A =0 in the message reflects the resolved limit at the time; when the guard fires, the query wants more than it is allowed. The fix is to make the query cheaper or to give it (and the process) more memory to work with.

Symptoms

• A query fails with the query requires more than -search.maxMemoryPerQuery ... bytes of memory.
• Heavy aggregations (sum, topk, quantile) over large series sets error, while narrower versions succeed.
• The same query works when few dashboards are open but fails under heavy concurrent load.
• Very wide selectors or large subqueries reliably trip the limit.
• Reducing the time range or adding a label matcher makes the query run.

Common Root Causes

• Heavy aggregation over a large series set — grouping or summarizing across tens of thousands of series needs a lot of working memory at once.
• High query concurrency — many simultaneous queries competing for the shared memory budget shrink what each one can use.
• Very wide selectors — matchers with little or no filtering pull an enormous series set into the query.
• Big subqueries — nested range subqueries (for example max_over_time( rate(m[5m])[1d:1m] )) expand the intermediate data materialized in memory.

How to diagnose

Gauge the query’s weight by counting the series it touches:

curl -s -G 'http://localhost:8481/select/0/prometheus/api/v1/series' \
  --data-urlencode 'match[]=<your_selector>' \
  --data-urlencode 'start=-1h' \
  | python3 -c 'import sys,json;print(len(json.load(sys.stdin)["data"]))'

Check the configured per-query limit, the allowed-memory percent, and current concurrency settings:

ps aux | grep -E 'vmselect|victoria-metrics' \
  | grep -oE '\-(search.maxMemoryPerQuery|memory.allowedPercent|search.maxConcurrentRequests)[^ ]*'

Watch memory and concurrency metrics while reproducing the query to see whether it is a single fat query or contention with others:

curl -s http://localhost:8481/metrics | grep -E \
  'vm_memory_intensive_queries_total|vm_concurrent_select_(current|capacity)'

If the failing query touches a huge series set, tightening filters is more effective than raising the budget.

Fixes

1. Narrow the label matchers and time range. The cheapest and safest fix — scope the query so it aggregates far fewer series and datapoints:

# Instead of aggregating across everything:
sum(rate(http_requests_total[5m]))

# Filter first, then aggregate:
sum by(service) (rate(http_requests_total{cluster="prod", env="live"}[5m]))

2. Raise -search.maxMemoryPerQuery — within total RAM. Give a legitimately heavy query more room, remembering the effective ceiling is bounded by -memory.allowedPercent and the node’s physical memory:

./vmselect -storageNode=vmstorage-1:8401,vmstorage-2:8401 \
  -search.maxMemoryPerQuery=1073741824

Do not set it near total RAM — the budget is shared across concurrent queries.

3. Lower -search.maxConcurrentRequests so each query gets more budget. Fewer in-flight queries means a larger share of the shared memory pool per query:

./vmselect -storageNode=vmstorage-1:8401,vmstorage-2:8401 \
  -search.maxConcurrentRequests=4

This trades throughput for per-query headroom; excess queries queue instead of failing.

4. Split the query. Break one wide query into several narrower ones (per cluster, per service, or over shorter ranges) and combine the results, so no single request exceeds the budget. For recurring heavy aggregations, precompute a smaller series with vmalert recording rules and query that instead.

What to watch out for

• The effective per-query budget is the minimum of -search.maxMemoryPerQuery and what -memory.allowedPercent leaves free — raising one without headroom in the other changes nothing.
• Raising the limit too high lets a single query starve others or OOM the node; size it against RAM and concurrency together, not in isolation.
• This guard is about memory, distinct from the sample-count guard (-search.maxSamplesPerQuery) — a query can be under the sample limit yet over the memory limit because of heavy aggregation.
• Lowering -search.maxConcurrentRequests helps per-query memory but reduces throughput; balance it against your real query load.
• Subqueries multiply intermediate data in memory; watch for nested [range:step] expressions when a query is unexpectedly heavy.

Related

• VictoriaMetrics Error Guide: OOM killed
• VictoriaMetrics Error Guide: max samples per query
• VictoriaMetrics Error Guide: max concurrent requests