🚨 VictoriaMetrics Error: 'dropping label ... too long value; -maxLabelValueLen=1024 exceeded' — Cause, Fix, and Troubleshooting Guide

Overview

VictoriaMetrics guards against oversized label values during ingestion. Any single label value longer than -maxLabelValueLen (default 1024 bytes) is dropped, and vmagent or vmstorage logs a warning identifying the metric and the offending label:

dropping label {__name__="app_request_duration_seconds", path="..."} with too long value for metric; -maxLabelValueLen=1024 exceeded

Note the wording: the label is dropped, not the whole sample — the metric still ingests, just without that label, which can silently break dashboards and alerts that depend on it. This limit exists because huge label values inflate index size, cardinality, and memory. It almost always signals an instrumentation mistake: something high-entropy has been stuffed into a label value that should be short and bounded.

Symptoms

• Repeated dropping label ... too long value warnings in vmagent/vmstorage logs.
• A label you expect (like path, url, or error) is missing from stored series.
• Queries and Grafana panels that group by the dropped label return unexpected or empty results.
• The warnings correlate with a specific exporter or application after a code change.
• Index/memory usage creeps up on metrics carrying near-limit label values.

Common Root Causes

• Full URLs or paths in a label — instrumentation putting raw request paths with query strings into a path or url label.
• Stack traces or error blobs in a label — an error label carrying a multi-line trace instead of a short code.
• Base64 blobs or serialized data — encoded payloads or tokens placed in a label value.
• A runaway label value — a request ID, session, or JSON document jammed into a single label.

How to diagnose

Confirm the current limit and find which exporter is emitting long values:

# Current configured limit
ps aux | grep -E 'vmagent|victoria-metrics' | grep -oE '\-maxLabelValueLen[^ ]*'
# Recent drop warnings, grouped by metric name
journalctl -u vmagent --since '1h ago' | grep 'too long value' | sort | uniq -c | sort -rn

Inspect the raw exporter output to see the actual value length before it reaches VictoriaMetrics:

# Look for suspiciously long label values in the scrape target
curl -s 'http://app-host:9100/metrics' \
  | awk '{ if (length($0) > 1024) print length($0), $0 }' | head

Fixes

1. Drop or truncate the offending label at ingest with metric_relabel_configs. Remove a label that should never have been high-entropy:

metric_relabel_configs:
  # Drop the runaway label entirely
  - action: labeldrop
    regex: "path"

Or truncate/normalize it to a bounded value with a capture:

metric_relabel_configs:
  # Keep only the first path segment, strip query strings
  - source_labels: [path]
    target_label: path
    regex: "(/[^/?]*).*"
    replacement: "$1"
    action: replace

2. Fix the exporter/instrumentation so it emits short, bounded label values — use a route template (/users/:id) instead of the concrete URL, and an error code instead of a full trace.

3. Raise -maxLabelValueLen only cautiously. A higher ceiling lets larger values through, inflating cardinality and memory; do it only when values are legitimately long and bounded:

./vmagent -maxLabelValueLen=2048 -remoteWrite.url=http://vminsert:8480/insert/0/prometheus/

What to watch out for

• The metric still ingests without the dropped label, so alerts that group by it can silently misfire — treat these warnings as real data loss.
• Raising the limit does not fix the root cause; a leaky label keeps growing cardinality until it hurts query latency and vmstorage RAM.
• Prefer labeldrop/replace at the vmagent layer so bad values never reach storage, rather than cleaning up after the fact.
• Watch for the same pattern across multiple exporters — a shared instrumentation library often puts URLs or traces into labels everywhere at once.

Related

• VictoriaMetrics Error Guide: labels count exceeds limit
• VictoriaMetrics Error Guide: high churn rate
• VictoriaMetrics Error Guide: max series