🚨 VictoriaMetrics Error Guide: 'the response from ... exceeds -promscrape.maxScrapeSize' — Right-Size the Scrape

Overview

vmagent (and single-node victoria-metrics when it scrapes directly) caps the size of a single scrape response. When a target returns a /metrics body larger than -promscrape.maxScrapeSize (default 16MB), the scrape is aborted and the target is marked down with this error on the /targets page and in the logs:

error when scraping "http://exporter:9100/metrics": the response from "http://exporter:9100/metrics" exceeds -promscrape.maxScrapeSize=16777216 bytes; either reduce the response size or increase -promscrape.maxScrapeSize

The limit is a guard: an unbounded scrape body can blow up vmagent’s memory as it parses and relabels every series at once. Rather than risk an OOM, vmagent drops the oversized scrape entirely — which means you lose all series from that target, not just the excess.

Symptoms

• A target flips to up == 0 on vmagent’s /targets page with “exceeds -promscrape.maxScrapeSize” as the last error.
• An entire exporter’s metrics vanish from dashboards at once (all-or-nothing, not partial gaps).
• The failure often appears suddenly after a deploy that added labels or a new subsystem’s metrics.
• vm_promscrape_scrapes_failed_total for the affected job increments on every scrape interval.
• The target’s /metrics endpoint is visibly large when fetched by hand (tens of MB).

Common Root Causes

• High-cardinality exporter output — a label like pod, container_id, path, or user_id explodes one metric into hundreds of thousands of series, inflating the response body.
• cAdvisor / kube-state-metrics on large nodes — dense per-container metrics on a busy node routinely push past 16MB.
• Aggregating exporters — a blackbox/pushgateway/federation endpoint that concatenates many jobs into one scrape.
• A metric leak in application code — unbounded label values (request IDs, full URLs, timestamps) added to a counter.
• Default limit left in place on a workload that has simply grown past it over time.

How to diagnose

Confirm the actual response size before touching config. Fetch the target the same way vmagent does and measure the bytes:

# Size of the raw scrape body
curl -s http://exporter:9100/metrics | wc -c

# Which metric names dominate the response
curl -s http://exporter:9100/metrics | grep -vE '^#' | cut -d'{' -f1 | sort | uniq -c | sort -rn | head -20

The uniq -c output shows which metric is producing the most series — that is almost always your cardinality culprit. Cross-check the running limit and per-job failures in vmagent:

# Current configured limit
ps aux | grep vmagent | grep -oE '\-promscrape.maxScrapeSize[^ ]*'

# Failed scrapes per job
curl -s http://localhost:8429/metrics | grep vm_promscrape_scrapes_failed_total

vmagent’s /targets page shows the last error and last scrape size per target; use it to confirm this specific error rather than a generic timeout.

Fixes

1. Cut the response size at the source (preferred). Drop the high-cardinality series with metric_relabel_configs so vmagent never has to hold them:

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["exporter:9100"]
    metric_relabel_configs:
      # Drop a noisy, unbounded metric entirely
      - source_labels: [__name__]
        regex: "node_scrape_collector_duration_seconds"
        action: drop
      # Strip a high-cardinality label instead of the whole metric
      - regex: "path"
        action: labeldrop

2. Raise the limit deliberately when the target is legitimately large and you have the memory headroom. Set it per vmagent, and size vmagent RAM accordingly:

./vmagent -promscrape.config=scrape.yml \
  -remoteWrite.url=http://vminsert:8480/insert/0/prometheus/api/v1/write \
  -promscrape.maxScrapeSize=32MB

3. Reduce what the exporter emits. For kube-state-metrics use --metric-allowlist/--metric-denylist; for cAdvisor disable unused collectors; for application code, remove unbounded labels so the metric stops growing.

4. Split the scrape. If one endpoint federates many jobs, scrape the underlying targets directly (smaller bodies each) instead of one giant federation endpoint.

What to watch out for

• Raising -promscrape.maxScrapeSize treats the symptom — an oversized scrape usually means a cardinality problem that will also strain vmstorage and vmselect downstream. Prefer trimming.
• The failure is all-or-nothing: until you fix it, you lose every series from that target, so treat it as an outage for that job, not a partial degradation.
• labeldrop/labeldelete can collapse previously-distinct series into duplicates — verify the metric still makes sense after stripping a label.
• Watch vm_promscrape_scrape_response_size_bytes (or the target’s last scrape size) so you catch a target creeping toward the limit before it trips.

Related

• VictoriaMetrics Error Guide: high churn rate
• VictoriaMetrics Error Guide: remote_write 429
• VictoriaMetrics Error Guide: too many open files