🚨 Prometheus Error Guide: 'body size limit exceeded' — Trim or Raise the Scrape Cap

Overview

When a target’s /metrics response is larger than the job’s configured body_size_limit, Prometheus aborts the read mid-stream, discards the entire scrape, and marks the target down:

ts=2026-07-06T10:03:57.421Z caller=scrape.go:1382 level=warn component="scrape manager" scrape_pool=app-metrics target=http://10.2.9.14:9102/metrics msg="Scrape failed" err="body size limit exceeded (limit: 10485760 bytes)"

The same reason surfaces on /api/v1/targets:

"health": "down",
"lastError": "body size limit exceeded (limit: 10485760 bytes)",
"scrapeUrl": "http://10.2.9.14:9102/metrics"

body_size_limit caps the number of bytes Prometheus will read from a single scrape response (default 0 = unlimited unless set). Unlike sample_limit, which counts series after parsing, body_size_limit is enforced on the raw response body as it streams in — so an oversized payload is rejected before it can be parsed or stored. As with other scrape limits, the whole scrape is dropped, not truncated, so up goes to 0.

Symptoms

• A target flips to down with body size limit exceeded (limit: N bytes) while the exporter process itself is healthy and reachable.
• curling the target’s /metrics returns a large response (tens of MB) that takes a while to complete.
• The failure appeared after an exporter upgrade, a new high-cardinality metric, or enabling extra collectors.
• up == 0 for the target, and any alerts on its series report no data / absent().
• Other, smaller targets in the same job scrape fine — only the fat ones fail.

Common Root Causes

1. A genuinely huge exporter — kube-state-metrics, cAdvisor, or a custom exporter emitting hundreds of thousands of lines, legitimately exceeding a conservative byte cap.
2. A cardinality explosion — an unbounded label (request IDs, raw URLs, UUIDs) multiplying series and inflating the response into the megabytes.
3. A body_size_limit set too low — a cautious global default applied uniformly to jobs of very different sizes.
4. Verbose metric help/type comments — exporters that emit long # HELP/# TYPE text per metric, bloating the body beyond what the series count alone suggests.
5. Accidental duplication — a misconfigured exporter exposing multiple registries or repeating metric families in one response.

Diagnostic Workflow

Find every target down for the byte-limit reason:

curl -s http://localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | select(.lastError|test("body size limit")) | [.scrapePool,.scrapeUrl,.lastError] | @tsv'

app-metrics  http://10.2.9.14:9102/metrics  body size limit exceeded (limit: 10485760 bytes)

Measure the actual response size the exporter is producing (bytes and line count):

curl -s http://10.2.9.14:9102/metrics | wc -c
curl -s http://10.2.9.14:9102/metrics | grep -vcE '^(#|$)'

14680320
118442

Rank the biggest metric families so you know what to drop:

curl -s http://10.2.9.14:9102/metrics \
  | grep -vE '^#' | awk '{print $1}' | sed 's/{.*//' | sort | uniq -c | sort -rn | head

Read the configured limit from the live config:

curl -s http://localhost:9090/api/v1/status/config | jq -r '.data.yaml' \
  | grep -B2 -A6 'job_name: app-metrics' | grep body_size_limit

The example scrape config that triggers it:

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: "app-metrics"
    body_size_limit: 10MB          # too small for a 14MB response
    static_configs:
      - targets: ["10.2.9.14:9102"]

Confirm the fix by dropping the noisiest family and reloading:

topk(10, count by (__name__) ({job="app-metrics"}))

Example Root Cause Analysis

An SRE set a global body_size_limit: 10MB as a defensive default. Weeks later a single app-metrics target went down with body size limit exceeded (limit: 10485760 bytes) while its peers stayed green.

curl … | wc -c showed the response was 14 MB and grep -vcE '^(#|$)' counted 118k lines. Ranking families revealed one custom metric, http_request_duration_seconds, carried a raw path label containing per-request IDs, producing ~90k unique series on its own.

The team did not just raise the cap. They added a metric_relabel_configs rule to rewrite the unbounded path into a bounded route template and drop the raw label, which shrank the response to 2.3 MB. Because that was a durable cardinality fix, the 10 MB cap could stay in place as a tripwire. Only after confirming the smaller body did they reload and watch the target return to up.

Had they simply raised body_size_limit to unlimited, the underlying cardinality bug would have kept growing until it hit TSDB memory limits instead — a worse failure later.

Prevention Best Practices

• Set body_size_limit on jobs as a byte tripwire sized with headroom over the real response size, especially for large exporters like kube-state-metrics and cAdvisor.
• Fix cardinality at the source: rewrite unbounded labels (paths, IDs, UUIDs) into bounded values with metric_relabel_configs rather than raising limits forever.
• Combine body_size_limit (bytes) with sample_limit (series) — they guard different failure modes and catch different explosions.
• Alert on scrape health (up == 0) correlated with the exporter being reachable, so a limit trip is distinguishable from a network outage.
• Re-evaluate byte and sample limits whenever an exporter is upgraded or new collectors are enabled.
• Prefer route templates and status classes in labels; never embed per-request identifiers.

Quick Command Reference

# Targets down on body_size_limit
curl -s http://localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | select(.lastError|test("body size limit")) | [.scrapeUrl,.lastError] | @tsv'

# Actual response size (bytes) and series count
curl -s http://10.2.9.14:9102/metrics | wc -c
curl -s http://10.2.9.14:9102/metrics | grep -vcE '^(#|$)'

# Biggest metric families in the payload
curl -s http://10.2.9.14:9102/metrics \
  | grep -vE '^#' | awk '{print $1}' | sed 's/{.*//' | sort | uniq -c | sort -rn | head

# Configured body_size_limit for the job
curl -s http://localhost:9090/api/v1/status/config | jq -r '.data.yaml' \
  | grep -A6 'job_name: app-metrics' | grep body_size_limit

# Validate + reload after trimming or raising the limit
promtool check config /etc/prometheus/prometheus.yml \
  && curl -s -XPOST http://localhost:9090/-/reload

Conclusion

body size limit exceeded means a target’s raw /metrics payload outgrew the byte cap you set, and Prometheus dropped the whole scrape to protect itself. The right response is diagnostic, not reflexive: measure the real response size, rank the fattest metric families, and decide whether the growth is legitimate or a cardinality bug. When a label is exploding, fix it with metric_relabel_configs at the source so the cap can stay as a tripwire; only raise body_size_limit when the volume is genuinely justified and sized with headroom. Paired with sample_limit, a byte cap keeps a single runaway exporter from quietly filling your TSDB — as long as you treat the error as an early warning rather than a nuisance to silence.