🚨 Prometheus Error Guide: 'label_limit exceeded' Target Scrape Rejected

Exact Error Message

label_limit exceeded and its siblings are scrape-time rejections. When a scrape_config sets label_limit, label_name_length_limit, or label_value_length_limit, Prometheus enforces those limits while parsing the target’s /metrics response. If any series breaks a limit, the entire scrape is discarded and the target is marked down with the error in its scrape status.

err="label_limit exceeded (got 41 labels, limit is 30)"

The related variants:

err="label_name_length_limit exceeded (got 312 bytes for label name, limit is 200)"
err="label_value_length_limit exceeded (got 2048 bytes for label value, limit is 1024)"
err="labels for metric \"http_requests_total\" exceeds the limit of 30 label(s)"

What the Error Means

These limits are guardrails against runaway cardinality. label_limit caps how many labels a single series may carry; label_name_length_limit and label_value_length_limit cap the byte length of any label name or value. They are configured per scrape job (or globally) and enforced at parse time.

When a target violates a limit, Prometheus does not drop just the bad series — it rejects the whole scrape, so the target shows as down and you lose every metric from it, not only the offending one. This is a behavioral safety feature: it stops a misbehaving exporter from polluting the TSDB with high-cardinality or oversized labels.

Common Causes

1. A scrape_config sets a limit that a target legitimately exceeds — the limit is too low for a verbose but valid exporter.
2. An exporter emitting too many labels per series — kube-state-metrics or cAdvisor stacking many label_* annotations onto each series.
3. Oversized label values — full URLs, stack traces, JSON blobs, or base64 IDs shoved into a label value.
4. IDs used as labels — request IDs, trace IDs, or user IDs creating long, high-cardinality values.
5. An accidental high-cardinality label — a new code path adds a label like query or path with unbounded content.
6. Global limits inherited unexpectedly — a limit set under global: applies to every job, surprising one verbose target.

How to Reproduce the Error

Set a deliberately low limit and point it at any moderately labeled exporter:

scrape_configs:
  - job_name: ksm
    label_limit: 8
    label_value_length_limit: 64
    static_configs:
      - targets: ["kube-state-metrics:8080"]

kube-state-metrics routinely emits more than eight labels per series, so the scrape is rejected immediately and the target flips to down after a reload.

Diagnostic Commands

Confirm the target is down and read the exact error from the targets API:

curl -s http://localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | select(.health=="down") | [.labels.job, .scrapeUrl, .lastError] | @tsv'

ksm   http://kube-state-metrics:8080/metrics   label_limit exceeded (got 14 labels, limit is 8)

Count labels per series directly on the target’s /metrics to find the worst offender:

# Max number of labels on any single series
curl -s http://kube-state-metrics:8080/metrics \
  | grep -oE '\{[^}]*\}' \
  | awk -F',' '{print NF}' | sort -rn | head -1

# Longest label value on the target
curl -s http://kube-state-metrics:8080/metrics \
  | grep -oE '="[^"]*"' | awk '{ print length, $0 }' | sort -rn | head -5

Inspect the limits actually configured for the job:

grep -nE 'label_limit|label_name_length_limit|label_value_length_limit' /etc/prometheus/prometheus.yml

Check the broader cardinality picture so you fix the right thing:

curl -s http://localhost:9090/api/v1/status/tsdb \
  | jq '.data.labelValueCountByLabelName[0:10]'

# Once the target is up, see which label has the most values
topk(10, count by (__name__)({job="ksm"}))

Step-by-Step Resolution

1. Decide: is the data legitimate or junk? If the extra labels are useful, raise the limit. If they are an ID, URL, or blob, drop them.

2a. To raise the limit deliberately, bump the specific limit on that job and reload. Do this knowingly — limits exist to protect memory.

scrape_configs:
  - job_name: ksm
    label_limit: 40
    label_value_length_limit: 2048
    static_configs:
      - targets: ["kube-state-metrics:8080"]

2b. To drop the offending label instead, add metric_relabel_configs so the label is removed before the limit is checked downstream — and so the scrape succeeds.

scrape_configs:
  - job_name: app
    static_configs: [{ targets: ["app:8080"] }]
    metric_relabel_configs:
      # Remove an unbounded "url" label that blows the value-length limit
      - action: labeldrop
        regex: url
      # Or truncate / replace a long ID label
      - source_labels: [trace_id]
        target_label: trace_id
        replacement: "redacted"

Note: metric_relabel_configs runs after the limit check in older versions; if relabeling alone does not clear the error, raise the limit slightly so the scrape parses, then let labeldrop shrink the stored series.

3. Fix the exporter where possible. A long URL or full path in a label is almost always an instrumentation bug — move it to a log line or an exemplar, not a label.

4. Reload and confirm the target is back up.

curl -s -X POST http://localhost:9090/-/reload
curl -s http://localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | select(.labels.job=="ksm") | .health'

Prevention and Best Practices

• Set sane global limits (e.g. label_limit: 32, label_value_length_limit: 1024) so a new exporter cannot quietly add a high-cardinality label.
• Treat label values as bounded enums — never IDs, URLs, timestamps, or free text.
• Alert on up == 0 joined with target lastError so a limit rejection surfaces as a down target, not silent data loss.
• Review new exporters’ /metrics before enabling limits in production so you size them correctly.
• Use metric_relabel_configs labeldrop as the first line of defense for known-noisy labels.

Related Errors

• Prometheus ‘scrape_sample_limit exceeded’ — a different limit. scrape_sample_limit caps the number of series (samples) a target may expose; label_limit caps the number/size of labels on each series. One is series count, the other is label count.
• Prometheus OOMKilled / high memory — uncontrolled labels are the usual road to an OOM.
• Prometheus ‘compaction failed’ — extreme cardinality makes compactions fragile.

Frequently Asked Questions

Is label_limit the same as scrape_sample_limit? No, and confusing them is common. scrape_sample_limit rejects a scrape that exposes too many series (samples) in total. label_limit rejects a scrape where a single series carries too many labels. A target can pass one and fail the other. See the scrape_sample_limit guide.

Why does one bad series take down the whole target? Limits are enforced at parse time, and Prometheus rejects the entire scrape rather than partially ingesting it. This is intentional — partial scrapes would create misleading gaps. Fix the offending series and the full scrape resumes.

Should I just raise the limit to make the error go away? Only if the labels are genuinely useful. Limits protect memory; raising them to swallow an exporter dumping URLs or IDs into labels trades a clear error for a slow OOM. Prefer labeldrop for junk labels.

Where do I set these limits? Per scrape_config, or under global: to apply to every job. Per-job settings override the global. Check both with grep -nE 'label_limit|length_limit' prometheus.yml.

Which label is breaking the limit? Count labels per series on the raw /metrics (grep -oE '\{[^}]*\}' | awk -F, '{print NF}') and find the longest values with the length sort shown above. The biggest line is your offender.