🚨 Prometheus Error Guide: 'dial tcp: i/o timeout' — Fix Scrape Timeouts

Overview

Prometheus marks a target DOWN and records this error on the target’s status page whenever a scrape cannot complete a TCP read or connect within the allotted time:

Get "http://10.0.0.5:9100/metrics": dial tcp 10.0.0.5:9100: i/o timeout

The same class of failure also appears mid-transfer, after the connection is established but the body never fully arrives:

Get "http://10.0.0.5:9100/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Both mean the scrape did not finish within scrape_timeout. The target may be reachable but slow, or the network path may be silently dropping packets. Until it resolves, up{} for that target is 0 and every alert and dashboard built on those metrics goes blind.

Symptoms

• The target shows DOWN in Status → Targets with a dial tcp ...: i/o timeout error.
• up{job="..."} == 0 for the affected instance, often flapping between 0 and 1.
• Scrape duration for the target sits near or above scrape_timeout right before it fails.
• Other targets on the same host or subnet may fail together (shared network path).
• Manual curl of /metrics from the Prometheus host is slow or hangs.

Common Root Causes

• The target is genuinely slow — the exporter takes longer than scrape_timeout to render /metrics (huge cardinality, expensive collectors, or a blocked backend).
• Firewall / security group drops — packets are silently dropped (not rejected), so the connection hangs until timeout instead of failing fast.
• DNS resolution latency — a slow resolver delays the dial phase, eating the timeout budget before bytes flow.
• Network saturation or packet loss on the path between Prometheus and the target.
• scrape_timeout too low for a legitimately large payload, especially node_exporter with many collectors or kube-state-metrics on big clusters.
• Target overloaded — CPU-starved exporter cannot serve the request in time.

Diagnostic Workflow

Start by reproducing the scrape from the Prometheus host to separate “target slow” from “network broken”:

time curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' \
  http://10.0.0.5:9100/metrics

If that hangs, test raw connectivity and DNS independently:

nc -vz -w 5 10.0.0.5 9100        # TCP reachability
dig +short exporter.internal      # is DNS itself slow/failing?
mtr -rwc 20 10.0.0.5              # packet loss / latency along the path

Check how close scrapes run to the timeout using Prometheus’s own meta-metrics:

scrape_duration_seconds{job="node"} 
  / on(job) group_left() scrape_timeout_seconds

Confirm the configured budget in the job:

scrape_configs:
  - job_name: node
    scrape_interval: 30s
    scrape_timeout: 10s        # must be <= scrape_interval
    static_configs:
      - targets: ["10.0.0.5:9100"]

If the exporter itself is the bottleneck, measure how long it takes to render, and how big the payload is:

curl -sS http://10.0.0.5:9100/metrics | wc -l

Example Root Cause Analysis

A node_exporter fleet started flapping after a systemd rollout added hundreds of units per host. curl of /metrics returned HTTP 200 but took 11.4s; scrape_timeout was 10s. The mtr path was clean with 0% loss, so the network was ruled out. The systemd collector was enumerating thousands of units, and rendering exceeded the budget.

The fix was two-fold: restrict the systemd collector to a unit allowlist (--collector.systemd.unit-include) to cut render time to 1.2s, and raise scrape_timeout to 15s on that job as headroom. up{} returned to a stable 1 and scrape_duration_seconds dropped well below the timeout.

Prevention Best Practices

• Keep scrape_timeout comfortably below scrape_interval, and size it against the target’s real p99 render time plus network margin.
• Alert on scrape_duration_seconds / scrape_timeout crossing ~0.8 so you catch creeping slowness before targets flap.
• Prefer reject over silent drop in firewalls for monitored ports so failures are fast and diagnosable.
• Trim expensive exporter collectors and high-cardinality endpoints rather than endlessly raising timeouts.
• Alert on up == 0 grouped by target, and separately on flapping (changes(up[15m]) > 3).

Quick Command Reference

# Reproduce the scrape with timing
time curl -sS -o /dev/null -w '%{time_total}s\n' http://TARGET:PORT/metrics

# Raw TCP + DNS + path checks
nc -vz -w 5 TARGET PORT
dig +short TARGET_HOST
mtr -rwc 20 TARGET

# Payload size (proxy for render cost)
curl -sS http://TARGET:PORT/metrics | wc -l

up == 0
scrape_duration_seconds
changes(up[15m]) > 3

Conclusion

dial tcp: i/o timeout is Prometheus telling you a scrape ran out of time — either the target is too slow or the network path is dropping traffic. Reproduce with a timed curl, split network from target with nc/mtr, then either fix the slow exporter (trim collectors, reduce cardinality) or grant realistic scrape_timeout headroom. Alerting on scrape duration relative to the timeout turns this from a recurring flap into an early warning.