🚨 VictoriaMetrics Error Guide: 'the number of matching time series exceeds -search.maxSeries' — Bound the Match Set

Overview

VictoriaMetrics caps how many distinct time series a single request may match. When a /api/v1/series lookup, label-values call, or query selects more series than -search.maxSeries (default 30000), vmselect (or single-node victoria-metrics) aborts the request with:

error when searching for time series: the number of matching time series exceeds -search.maxSeries=30000; either increase -search.maxSeries or narrow down the query with the more specific label filters

This protects vmselect from loading a huge series set into memory to satisfy one metadata or query request. It is a read-path guard, distinct from the ingest-side series ceiling — it fires on lookups, most often from Grafana’s variable/label auto-complete and from broad {__name__=~".+"}-style selectors.

Symptoms

• Grafana template variables fail to populate, or a dashboard panel errors with “the number of matching time series exceeds -search.maxSeries”.
• /api/v1/series and /api/v1/label/<name>/values calls return HTTP errors on busy tenants.
• The error appears when someone opens a dashboard with a wide-open label dropdown (e.g. label_values(node_cpu_seconds_total, instance) across everything).
• Explore/autocomplete in vmui or Grafana is slow then fails on high-cardinality metrics.
• The same query works fine over a narrow time range or a filtered tenant, but fails when broadened.

Common Root Causes

• Unbounded matchers — a selector like {__name__=~".+"} or {job=~".+"} that pulls in every series.
• High-cardinality labels in a dropdown — Grafana label_values() over pod, container_id, uuid, or instance on a large fleet.
• A metric that has quietly exploded — a label leak pushed one metric past 30k series, so any lookup touching it trips the limit.
• Wide time ranges — a long range accumulates more distinct series (churn) than a short one, tipping a borderline query over the edge.
• Default limit on a large deployment — 30k is fine for a small setup but low for a big multi-tenant cluster.

How to diagnose

Find which metrics carry the most series using the built-in TSDB status endpoint — this points straight at the cardinality source:

# Top metric names and label pairs by series count
curl -s 'http://localhost:8481/select/0/prometheus/api/v1/status/tsdb' | head -60

Reproduce the offending lookup and count what it matches over the same range:

# How many series does this selector actually match?
curl -s -G 'http://localhost:8481/select/0/prometheus/api/v1/series' \
  --data-urlencode 'match[]={__name__=~"node_.+"}' \
  --data-urlencode 'start=-1h' | python3 -c 'import sys,json;print(len(json.load(sys.stdin)["data"]))'

Check the currently configured limit:

ps aux | grep -E 'vmselect|victoria-metrics' | grep -oE '\-search.maxSeries[^ ]*'

If the count is near or above 30k, the fix is to narrow the match set — not just to raise the number.

Fixes

1. Narrow the matcher (preferred). Add specific labels so the lookup touches far fewer series:

# Instead of a wide-open dropdown source:
label_values(node_cpu_seconds_total, instance)

# Scope it to the current dashboard's job/cluster variables:
label_values(node_cpu_seconds_total{job="node", cluster="$cluster"}, instance)

2. Chain Grafana variables so each dropdown filters by the ones above it ($cluster -> $job -> $instance), keeping every label_values() query small.

3. Raise the limit deliberately when the workload legitimately needs it and vmselect has the memory:

./vmselect -storageNode=vmstorage-1:8401,vmstorage-2:8401 \
  -search.maxSeries=100000

Set it in proportion to vmselect RAM; a higher ceiling means bigger metadata responses held in memory.

4. Attack the underlying cardinality. If one metric ballooned from a label leak, drop or trim that label at ingest with metric_relabel_configs (see the high-churn guide), so lookups shrink at the source.

What to watch out for

• Raising -search.maxSeries masks a cardinality problem that will also hurt query latency and vmstorage memory — narrow the matchers first, raise the limit second.
• This limit is for series lookups; a query that returns too many datapoints trips a different guard (too many points) — don’t confuse the two when reading errors.
• Grafana’s default “All” and multi-value template variables are a frequent trigger; prefer scoped, chained variables over wide-open dropdowns.
• Watch /api/v1/status/tsdb periodically so a metric creeping toward high cardinality is caught before every dashboard on it starts failing.

Related

• VictoriaMetrics Error Guide: max unique timeseries
• VictoriaMetrics Error Guide: too many points
• VictoriaMetrics Error Guide: labels count exceeds limit