🚨 VictoriaMetrics Error: '-search.maxTagValues=100000 label values found' — Cause, Fix, and Troubleshooting Guide

Overview

The label-values API (/api/v1/label/<name>/values) returns every distinct value a label takes. To keep one such lookup from loading a massive list into memory, vmselect (or single-node victoria-metrics) caps the result with -search.maxTagValues (default 100000). When a label has more distinct values than that, the request is refused:

error when searching for label values: -search.maxTagValues=100000 label values found; either narrow down the query with more specific filters or increase -search.maxTagValues

This is a read-path cardinality guard, cousin to -search.maxSeries and -search.maxTagKeys. It almost always fires on Grafana template-variable dropdowns built from a high-cardinality label — pod, container_id, uuid, instance — queried with no filters. The error is telling you the label itself is too wide, not that VictoriaMetrics is broken.

Symptoms

• A Grafana template variable that uses label_values(<label>) fails to populate and shows an error.
• /api/v1/label/<name>/values returns the error for one specific label while other labels resolve fine.
• The dropdown works when scoped to a job/cluster but fails when opened wide-open across everything.
• vmselect logs the -search.maxTagValues=100000 label values found line at the time the dashboard loads.
• The failing label is one known to be high-cardinality (per-pod, per-request, per-UUID identifiers).

Common Root Causes

• A very high-cardinality label queried without filters — asking for all values of pod or uuid across the whole dataset.
• A Grafana variable dropdown over an unbounded label — label_values(container_id) with no upstream variables constraining it.
• A label leak — a value that should be bounded (an ID, a URL, a hash) got attached as a label, inflating its distinct-value count past the ceiling.

How to diagnose

Find which labels carry the most distinct values using the TSDB status endpoint — it points straight at the offending label:

# Top labels by value count (seriesCountByLabelName / labelValueCountByLabelName)
curl -s 'http://localhost:8481/select/0/prometheus/api/v1/status/tsdb' | head -80

Reproduce the dropdown’s lookup and see how many values it really matches. A filtered form counts far fewer:

# Unbounded: likely trips the limit
curl -s -G 'http://localhost:8481/select/0/prometheus/api/v1/label/pod/values' \
  -o /dev/null -w '%{http_code}\n'

# Scoped with a matcher + narrow range: bounded result
curl -s -G 'http://localhost:8481/select/0/prometheus/api/v1/label/pod/values' \
  --data-urlencode 'match[]={namespace="prod", job="api"}' \
  --data-urlencode 'start=-1h' | python3 -c 'import sys,json;print(len(json.load(sys.stdin)["data"]))'

Check the configured ceiling:

ps aux | grep -E '[v]mselect|[v]ictoria-metrics' | grep -oE '\-search.maxTagValues[^ ]*'

Fixes

1. Scope the lookup with matchers and a narrow time range. Constrain the label-values query so it touches only the relevant series:

# Instead of a wide-open dropdown source:
label_values(pod)

# Scope it to the dashboard's namespace/job and a recent window:
label_values(kube_pod_info{namespace="$namespace", job="kube-state-metrics"}, pod)

2. Chain Grafana variables so each dropdown filters by the ones above it ($namespace -> $job -> $pod), keeping every label_values() result small:

# $pod depends on the already-selected $namespace
label_values(kube_pod_info{namespace="$namespace"}, pod)

3. Raise the limit only with memory headroom. If the workload legitimately needs a larger result set, bump the value-count and key-count ceilings together and size them against vmselect RAM:

./vmselect -storageNode=vmstorage-1:8401,vmstorage-2:8401 \
  -search.maxTagValues=300000 \
  -search.maxTagKeys=100000

4. Attack the underlying cardinality. If a label leak inflated the value count, drop or trim that label at ingest with metric_relabel_configs so the label stops exploding at the source:

metric_relabel_configs:
  - action: labeldrop
    regex: (uuid|container_id|request_id)

What to watch out for

• Raising -search.maxTagValues masks a cardinality problem that also inflates memory and slows every lookup on that label — scope the query first, raise the limit second.
• -search.maxTagValues (values of one label) is distinct from -search.maxTagKeys (number of label names) and -search.maxSeries (matching series) — read the error to know which guard fired.
• Grafana “All”/multi-value variables over a high-cardinality label are a frequent trigger; prefer scoped, chained dropdowns.
• Watch /api/v1/status/tsdb so a label creeping toward high cardinality is caught before every dashboard using it starts failing.

Related

• VictoriaMetrics Error Guide: max series
• VictoriaMetrics Error Guide: max unique timeseries
• VictoriaMetrics Error Guide: high churn rate