🚨 Prometheus Error Guide: 'connection reset by peer' — Fix Scrape Resets

Overview

Prometheus records this on a target’s status page when the TCP connection to the target is forcibly closed mid-scrape by the other end:

Get "http://10.0.0.9:8080/metrics": read tcp 10.0.0.2:52344->10.0.0.9:8080: read: connection reset by peer

A related variant appears when the reset lands during the write phase:

Get "http://10.0.0.9:8080/metrics": write tcp 10.0.0.2:52344->10.0.0.9:8080: write: connection reset by peer

An RST means the peer (the exporter, or a proxy/load balancer in front of it) tore the connection down rather than completing the response. The scrape fails, up{} goes to 0, and no samples land — but unlike a refused connection, something did answer, then aborted. That points at the target or an intermediary, not basic reachability.

Symptoms

• Target DOWN/flapping with read: connection reset by peer (or write: ... connection reset by peer).
• up == 0 intermittently, correlated with target restarts or high load.
• The exporter or app logs show a crash, panic, or OOM at scrape time.
• A proxy/load balancer or service mesh sidecar sits between Prometheus and the target.
• Resets cluster around large payloads or long-running scrapes.

Common Root Causes

• Exporter crash/OOM mid-response — the process dies while streaming /metrics, sending an RST.
• Idle/keep-alive timeout on a proxy — an LB, ingress, or mesh sidecar closes a connection Prometheus reused.
• Body-size or header limits on a proxy that aborts oversized metric responses.
• Connection limits — the target caps concurrent connections and resets extras (HA replicas both scraping).
• MTU / network middlebox dropping or resetting long-lived flows.
• App restarts during rollout — pods terminating between accept and response.

Diagnostic Workflow

Reproduce repeatedly to see if resets are consistent or load-shaped:

for i in $(seq 1 10); do
  curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' \
    http://10.0.0.9:8080/metrics || echo "reset"
done

Check whether the target process is crashing or being OOMKilled at scrape time:

# Kubernetes
kubectl -n app describe pod api-xyz | grep -A5 -iE 'last state|reason|restart|oomkilled'
kubectl -n app logs api-xyz --previous | tail -n 40

Confirm the payload size and whether a proxy is in the path:

curl -sS http://10.0.0.9:8080/metrics | wc -c      # bytes; large bodies stress proxies
curl -sS -D - -o /dev/null http://10.0.0.9:8080/metrics | grep -i 'server\|via\|x-envoy'

If a body-size limit is suspected, note that Prometheus can cap it too — check the job config:

scrape_configs:
  - job_name: api
    body_size_limit: 0        # 0 = unlimited; a low value here can also abort large bodies
    static_configs:
      - targets: ["10.0.0.9:8080"]

Correlate resets with target restarts:

changes(up{job="api"}[15m])
resets(process_start_time_seconds{job="api"}[15m])

Example Root Cause Analysis

An application’s /metrics endpoint intermittently reset scrapes with read: connection reset by peer. The up{} flaps lined up exactly with pod restarts visible in changes(process_start_time_seconds[15m]). The app exposed metrics on the same process that served traffic, and under memory pressure it was OOMKilled — killing the connection mid-response and emitting an RST.

kubectl logs --previous confirmed OOMKilled. The durable fix was to raise the pod’s memory limit and reduce a memory leak in the app, which stopped the restarts. As defense in depth, metrics were moved behind a lightweight endpoint less coupled to request-handling memory, so a busy app no longer took the scrape down with it. Resets ceased.

Prevention Best Practices

• Keep exporters/apps healthy: prevent OOMKills and crashes that reset in-flight scrapes (memory limits, leak fixes).
• Align proxy/LB/mesh idle and keep-alive timeouts to be longer than the scrape interval so reused connections aren’t torn down.
• Ensure proxy body-size and header limits exceed your largest /metrics payload.
• Avoid pointing multiple scrapers at a connection-capped target; front it with an exporter that tolerates concurrency.
• Alert on up == 0 plus changes(process_start_time_seconds[15m]) to attribute resets to restarts quickly.

Quick Command Reference

# Repeated scrape to expose intermittent resets
for i in $(seq 1 10); do curl -sS -o /dev/null -w '%{http_code}\n' http://TARGET:PORT/metrics || echo reset; done

# Crash / OOM inspection (Kubernetes)
kubectl -n NS describe pod POD | grep -A5 -iE 'last state|oomkilled|restart'
kubectl -n NS logs POD --previous | tail -n 40

# Payload size + proxy detection
curl -sS http://TARGET:PORT/metrics | wc -c

up == 0
changes(process_start_time_seconds[15m])

Conclusion

connection reset by peer means the target — or a proxy in front of it — answered and then aborted the connection. It is rarely a Prometheus-side problem: look for crashing/OOMKilled exporters, proxy keep-alive and body-size limits, or connection caps. Correlating resets with target restarts usually points straight at the cause, and stabilizing the target ends the flapping.