🚨 VictoriaMetrics Error Guide: 'the number of unique series exceeds -search.maxUniqueTimeseries' — Cut Query Cardinality

Overview

When a MetricsQL query would need to scan more distinct time series than the configured ceiling, VictoriaMetrics aborts it instead of loading them all into memory. The query returns HTTP 422/503 with:

error when executing query="...": cannot select more than -search.maxUniqueTimeseries=300000 unique time series during the selected time range; possible solutions: to increase -search.maxUniqueTimeseries; to reduce the time range for the query; to add more specific label filters

This guard protects vmselect (or a single node) from an out-of-memory crash triggered by an unbounded cardinality query. It signals that either the query is too broad or the underlying metric has a cardinality problem.

Symptoms

• A specific Grafana panel or vmalert rule fails while narrower queries succeed.
• The error names a concrete limit (default 300000) and the offending time range.
• Wide selectors like {__name__=~".+"} or metrics with a user_id/request_id label trigger it.
• Widening the dashboard time range makes a previously-working query start failing.
• Memory usage on vmselect spikes just before the query is aborted.

Common Root Causes

• Overly broad label matchers — queries with no or loose filters fan out across hundreds of thousands of series.
• High-cardinality labels on the metric — labels such as uuid, pod_hash, session_id, or full URLs multiply series counts.
• Large time ranges — a wide range covers more churned series (see churn rate), inflating the unique count.
• Regex matchers that match everything — =~".*" selects every series for a metric.
• Aggregations over raw high-cardinality data — sum without(...) still has to scan every underlying series first.
• Limit set too low for a legitimately large but healthy dataset.

Diagnostic Workflow

First identify what is driving cardinality using the built-in cardinality explorer in vmui (Cardinality tab) or the TSDB status API:

curl -s 'http://localhost:8428/api/v1/status/tsdb' | head
# Cluster mode:
curl -s 'http://vmselect:8481/select/0/prometheus/api/v1/status/tsdb'

This returns the top metric names by series count and the labels with the most values. Count series for a specific selector before running the full query:

curl -s 'http://localhost:8428/api/v1/series/count' \
  --data-urlencode 'match[]=http_requests_total'

In vmui, estimate how many series a matcher touches with MetricsQL:

count(count by (__name__) ({__name__=~"http_.+"}))

Find the label blowing up cardinality:

count(count by (le, path, user_id) (http_request_duration_seconds_bucket))

Check the active limit on the node:

ps aux | grep -oE '\-search.maxUniqueTimeseries[^ ]*'

Example Root Cause Analysis

A latency dashboard panel querying histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) worked at a 1-hour range but failed at 7 days with cannot select more than -search.maxUniqueTimeseries=300000 unique time series. The cardinality explorer showed http_request_duration_seconds_bucket carried a path label populated with raw request URLs (including query strings), producing millions of series over a week as endpoints churned.

The real fix was at instrumentation: the application was changed to record a normalized route template (/users/:id) instead of the full URL, collapsing the label to a few dozen values. As a stopgap, a metric_relabel_config on the collector dropped the raw path label, and a recording rule precomputed the p99 so the dashboard read a low-cardinality series. The limit was left at its default because the root cause was fixed, not masked.

Prevention Best Practices

• Never put unbounded values (IDs, UUIDs, full URLs, emails) in labels; normalize them at instrumentation time.
• Use metric_relabel_configs in vmagent/Prometheus to drop or aggregate high-cardinality labels before ingestion.
• Precompute expensive wide queries into recording rules so dashboards read low-cardinality series.
• Add specific label matchers to every dashboard query rather than relying on broad regexes.
• Watch the cardinality explorer and vm_cache_entries{type="storage/hour_metric_ids"} for growth trends.
• Raise -search.maxUniqueTimeseries only after confirming the cardinality is legitimate and RAM can hold it.

Quick Command Reference

# Top metrics and labels by cardinality
curl -s 'http://localhost:8428/api/v1/status/tsdb'

# Count series matching a selector
curl -s 'http://localhost:8428/api/v1/series/count' \
  --data-urlencode 'match[]=http_request_duration_seconds_bucket'

# Which label explodes cardinality (run in vmui)
# count(count by (path) (http_request_duration_seconds_bucket))

# Raise the limit (only if cardinality is legitimate)
./victoria-metrics-prod -search.maxUniqueTimeseries=1000000

Conclusion

This limit is a safety valve against a cardinality explosion, not a nuisance to bump. When it fires, use the cardinality explorer and /api/v1/status/tsdb to find the metric and label driving the series count. In the vast majority of cases the answer is to fix instrumentation or relabel away an unbounded label, then precompute wide queries with recording rules. Only raise -search.maxUniqueTimeseries once you have confirmed the data is genuinely large and healthy and your vmselect memory can support it.