🚨 Prometheus Error Guide: 'unexpected EOF' — Fix Truncated Scrapes

Overview

Prometheus records this on the target status page when the metrics response ends before the exposition body is complete — the connection closed, or the payload was cut off, mid-stream:

Get "http://10.0.0.11:2112/metrics": unexpected EOF

When the body is gzip-encoded and truncated, the parser reports the compression layer instead:

Get "http://10.0.0.11:2112/metrics": gzip: invalid header

Either way the scrape is discarded: Prometheus will not ingest a partial exposition, so up{} goes to 0. The target answered and started sending, but the response never finished — pointing at a timeout mid-body, a crashing exporter, a truncating proxy, or a content-encoding mismatch rather than a connectivity problem.

Symptoms

• Target DOWN/flapping with unexpected EOF (or gzip: invalid header / unexpected EOF from the gzip reader).
• up == 0 correlated with large /metrics payloads or scrapes that run near scrape_timeout.
• Manual curl sometimes returns partial output or the byte count varies between requests.
• A proxy, ingress, or mesh sidecar sits between Prometheus and the target.
• The exporter logs a panic or gets killed while rendering metrics.

Common Root Causes

• Timeout mid-body — scrape_timeout fires after headers but before the full body arrives, truncating the read.
• Exporter crash/panic while rendering — the process dies partway through streaming exposition.
• Proxy/LB response truncation — an intermediary with a response-size or time limit cuts the body.
• Content-encoding mismatch — the target sets Content-Encoding: gzip but sends non-gzip (or truncated gzip) data.
• Chunked transfer cut short — a middlebox drops the final chunk.
• Very large exposition approaching a body-size limit that aborts the transfer.

Diagnostic Workflow

Fetch the endpoint several times and watch for varying or short byte counts:

for i in $(seq 1 5); do
  curl -sS http://10.0.0.11:2112/metrics | wc -c
done

Check whether the body claims gzip and whether it decodes cleanly:

curl -sS -D - -o /tmp/m.out http://10.0.0.11:2112/metrics | grep -i 'content-encoding\|content-length'
file /tmp/m.out
gzip -t /tmp/m.out 2>&1 || echo "not valid gzip / truncated"

See how close the scrape runs to the timeout and whether it’s payload-bound:

scrape_configs:
  - job_name: app
    scrape_timeout: 10s
    body_size_limit: 0        # ensure a low limit isn't truncating large bodies
    static_configs:
      - targets: ["10.0.0.11:2112"]

scrape_duration_seconds{job="app"}
scrape_samples_scraped{job="app"}      # dips/zeros when bodies are cut short

Confirm the exporter isn’t dying mid-render:

kubectl -n app logs app-xyz --previous 2>/dev/null | tail -n 30
journalctl -u my-exporter --since '15 min ago' | grep -iE 'panic|signal|killed'

Example Root Cause Analysis

A custom Go exporter with a large, high-cardinality /metrics body began failing with unexpected EOF during peak hours. curl | wc -c returned a full ~9 MB body when the host was idle, but truncated counts under load. scrape_duration_seconds showed scrapes reaching the 10s scrape_timeout; when the timeout fired mid-body, Prometheus read a partial response and reported unexpected EOF.

The endpoint was CPU-bound rendering millions of series and simply couldn’t finish within 10s under load. The fix combined two changes: reduce the exposition size via metric_relabel_configs to drop unused high-cardinality series, and raise scrape_timeout to 20s as headroom. Render time dropped well under the budget and the EOFs stopped.

Prevention Best Practices

• Size scrape_timeout against the target’s worst-case render time so the body always completes before the deadline.
• Keep exposition payloads lean (drop unused high-cardinality series) so large-body truncation and slow renders don’t occur.
• Ensure proxies/LBs/mesh sidecars have response time and size limits above your largest /metrics response.
• Verify content-encoding correctness — only advertise gzip if the body is genuinely gzip.
• Alert on up == 0 alongside scrape_duration_seconds nearing the timeout to catch truncation before it flaps.

Quick Command Reference

# Detect varying/truncated bodies
for i in $(seq 1 5); do curl -sS http://TARGET:PORT/metrics | wc -c; done

# Encoding + gzip integrity
curl -sS -D - -o /tmp/m.out http://TARGET:PORT/metrics | grep -i 'content-encoding'
gzip -t /tmp/m.out 2>&1 || echo "truncated/invalid gzip"

# Exporter crash check
kubectl -n NS logs POD --previous | tail -n 30

up == 0
scrape_duration_seconds
scrape_samples_scraped

Conclusion

unexpected EOF means the metrics response was cut off before the exposition finished — most often a scrape that timed out mid-body, a crashing exporter, or a truncating proxy. Reproduce with repeated curl byte counts, check encoding integrity, and either give the scrape enough time or shrink the payload. Leaner exposition and realistic timeouts keep large /metrics endpoints parsing cleanly.