🚨 VictoriaMetrics Error Guide: 'labels count exceeds the limit' — Trim Per-Series Labels

Overview

VictoriaMetrics caps how many labels a single time series may carry. When an incoming sample has more labels than allowed, the extra labels are dropped and a warning is logged (per-metric, rate-limited):

the number of labels in the time series {__name__="app_http_requests_total", ...} exceeds -maxLabelsPerTimeseries=30; either reduce the number of labels for this metric or increase -maxLabelsPerTimeseries command-line flag value

The limit (default 30) is a guard against pathological instrumentation that attaches dozens of labels to every sample, which bloats the index and memory. When it trips, VictoriaMetrics keeps the series but silently truncates its labels, so data can become ambiguous or collide.

Symptoms

• Logs repeatedly warn the number of labels ... exceeds -maxLabelsPerTimeseries.
• Series that should be distinct merge together because their differentiating labels were dropped.
• Queries return fewer series than expected for a heavily-labeled metric.
• The warning names one or a few metrics coming from a specific exporter or app.
• vmagent logs the same when it enforces the limit before forwarding.

Common Root Causes

• Over-labeled instrumentation — an application attaching many dimensions (region, az, cluster, tenant, version, commit, handler, method, …) to one metric.
• Injected labels stacking up — external labels, target relabeling, and Kubernetes SD labels combining to push past 30.
• Debug/trace labels in production — labels meant for local debugging left enabled at scale.
• Duplicated dimensions — the same information encoded in several labels.
• Third-party exporters — an exporter that emits wide label sets by default.
• Limit set too low for a legitimately rich but bounded metric.

Diagnostic Workflow

Identify which metric is over-labeled from the warning, then inspect a raw sample to count labels. Query the series API for that metric:

curl -s 'http://localhost:8428/api/v1/series' \
  --data-urlencode 'match[]=app_http_requests_total' | head

Count the labels on a returned series to see how far over the limit it is. In vmui, list the label names attached to the metric with MetricsQL:

count(count by (__name__) (app_http_requests_total))

Check which labels are present and whether any are redundant:

curl -s 'http://localhost:8428/api/v1/labels' \
  --data-urlencode 'match[]=app_http_requests_total'

See the active limit and how often it is tripping:

ps aux | grep -oE '\-maxLabelsPerTimeseries[^ ]*'
curl -s http://localhost:8428/metrics | grep vm_metrics_with_dropped_labels_total

A rising vm_metrics_with_dropped_labels_total confirms labels are actively being truncated.

Example Root Cause Analysis

An application team noticed some series merging unexpectedly in Grafana. victoria-metrics logs showed the number of labels ... exceeds -maxLabelsPerTimeseries=30 for app_http_requests_total, and vm_metrics_with_dropped_labels_total was climbing. Inspecting a raw sample via /api/v1/series revealed 34 labels: the app added handler, method, status, region, az, cluster, tenant, version, git_commit, plus a dozen Kubernetes SD labels, and target relabeling piled on more. Labels beyond the 30th were being dropped in nondeterministic order, occasionally cutting the status label and merging 200s with 500s.

The fix was metric_relabel_configs in vmagent to labeldrop the debug-only and redundant labels (git_commit, duplicate az/region variants) before ingestion, bringing the series to 22 stable labels. Because the remaining labels were all legitimately needed, the default limit was left in place — the goal was fewer, meaningful labels, not a higher ceiling.

Prevention Best Practices

• Keep per-series labels lean and intentional; audit any metric approaching the -maxLabelsPerTimeseries default of 30.
• Use metric_relabel_configs with labeldrop to strip debug, redundant, or duplicate labels at the vmagent/collector.
• Avoid stacking external labels, SD labels, and relabeling outputs without reviewing the combined total.
• Never encode the same dimension in multiple labels.
• Alert on vm_metrics_with_dropped_labels_total so silent truncation is caught immediately.
• Raise -maxLabelsPerTimeseries only after confirming every label is genuinely needed and cardinality stays bounded.

Quick Command Reference

# Inspect the over-labeled series
curl -s 'http://localhost:8428/api/v1/series' \
  --data-urlencode 'match[]=app_http_requests_total'

# List labels on the metric
curl -s 'http://localhost:8428/api/v1/labels' \
  --data-urlencode 'match[]=app_http_requests_total'

# Confirm labels are being dropped
curl -s http://localhost:8428/metrics | grep vm_metrics_with_dropped_labels_total

# vmagent relabel snippet to drop excess labels
# metric_relabel_configs:
# - action: labeldrop
#   regex: git_commit|debug_.*|az_alias

# Raise the limit only if labels are all legitimate
./victoria-metrics-prod -maxLabelsPerTimeseries=40

Conclusion

“labels count exceeds the limit” means a metric is carrying more dimensions than VictoriaMetrics will index, and the overflow labels are being dropped — which can silently merge series that should stay distinct. Find the metric from the warning, inspect it via /api/v1/series, and trim redundant or debug labels with relabeling at ingestion. Reserve raising -maxLabelsPerTimeseries for the rare case where every one of those 30-plus labels is truly load-bearing.