🚨 VictoriaMetrics Error: 'cannot select more than -search.maxSamplesPerQuery samples' — Cause, Fix, and Troubleshooting Guide

Overview

To keep a single request from reading an unbounded amount of data, vmselect (or single-node victoria-metrics) counts how many raw samples a query would need to scan. When that count exceeds -search.maxSamplesPerQuery, the request is rejected before it can load everything into memory:

cannot select more than -search.maxSamplesPerQuery=1000000000 samples; possible solutions: to increase -search.maxSamplesPerQuery; to reduce the time range for the query; to use more specific label filters

The sample count is roughly the number of matched series multiplied by the number of datapoints each contributes over the time range. So a query trips this limit either by touching too many series (high cardinality) or by covering too long a range at too fine a resolution — usually both. The error itself lists the three levers: filter harder, shorten the range, or raise the limit. The first two are almost always the right move.

Symptoms

• A query or dashboard panel fails with cannot select more than -search.maxSamplesPerQuery ... samples.
• Wide rate() / increase() expressions over many series error, while the same expression on one instance works.
• An export or “all data” query over a long window fails immediately.
• The query succeeds over the last hour but fails when expanded to days or weeks.
• Broad selectors like {__name__=~".+"} or {job=~".+"} reliably trip the limit.

Common Root Causes

• Wide time range times high cardinality — a long window multiplied by many matched series produces a huge sample count.
• Large lookbacks over many series — rate(metric[1h]) or increase(metric[1d]) across a big fleet scans far more samples than a short window.
• Unbounded selectors — a query with no meaningful label matchers pulls in every series that shares the metric name (or all of them).
• Export-style queries — dumping raw data over a long period is inherently sample-heavy and is exactly what this guard limits.

How to diagnose

Estimate the two multipliers separately. First, count how many series the selector matches:

curl -s -G 'http://localhost:8481/select/0/prometheus/api/v1/series' \
  --data-urlencode 'match[]=<your_selector>' \
  --data-urlencode 'start=-1h' \
  | python3 -c 'import sys,json;print(len(json.load(sys.stdin)["data"]))'

Multiply that series count by the datapoints per series over your range (range in seconds divided by scrape interval). If the product approaches the limit, the query is genuinely too broad. Check the configured limit in use:

ps aux | grep -E 'vmselect|victoria-metrics' | grep -oE '\-search.maxSamplesPerQuery[^ ]*'

Find which metrics dominate cardinality so you know where to add filters:

curl -s 'http://localhost:8481/select/0/prometheus/api/v1/status/tsdb' | head -40

If the sample count is near the limit because of cardinality, narrow the matchers rather than raising the number.

Fixes

1. Add specific label matchers. Scope the query to the series you actually need so the sample count drops sharply:

# Instead of scanning every job:
rate(http_requests_total[5m])

# Filter to one service and cluster:
rate(http_requests_total{job="api", cluster="prod"}[5m])

2. Reduce the time range or increase the step. A shorter range or a coarser step means fewer datapoints per series. In Grafana, narrow the dashboard window or set Max data points/Min step on the panel; over the API, raise step on query_range.

3. Raise -search.maxSamplesPerQuery only with RAM headroom. If the workload legitimately needs it and vmselect has memory to spare, lift the ceiling deliberately:

./vmselect -storageNode=vmstorage-1:8401,vmstorage-2:8401 \
  -search.maxSamplesPerQuery=2000000000

A higher limit means a single query can pull more data into memory, so size it against available RAM and query concurrency.

4. Pre-aggregate with recording rules or stream aggregation. For dashboards that repeatedly scan the same wide set, precompute a smaller series (for example a per-cluster rollup) via vmalert recording rules or vmagent stream aggregation, then query the rollup instead of the raw metric.

What to watch out for

• Raising -search.maxSamplesPerQuery treats the symptom; a query that needs a billion-plus samples is usually too broad and will also be slow and memory-hungry even if it succeeds.
• This limit counts samples (series times datapoints); the too many points and -search.maxSeries guards count different things — read the exact error before deciding which lever to pull.
• Long lookbacks inside rate()/increase() multiply the sample count fast on large fleets; keep windows as short as the rule allows.
• Precomputing with recording rules cuts both this error and query latency, so prefer it over a bigger limit for recurring dashboard queries.

Related

• VictoriaMetrics Error Guide: too many points
• VictoriaMetrics Error Guide: max unique timeseries
• VictoriaMetrics Error Guide: max memory per query