🚨 Telegraf Error Guide: '[inputs.http] context deadline exceeded' — Fix HTTP Input Timeouts

Overview

The http input fetches a URL each interval and parses the body into metrics. When the target endpoint does not return response headers before the plugin’s timeout elapses, Go’s HTTP client cancels the request and Telegraf logs a context deadline error:

2026-07-12T12:00:00Z E! [inputs.http] Error in plugin: Get "https://api.example.com/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

A related form appears when the connection is established but the body stops streaming mid-read:

E! [inputs.http] Error in plugin: Get "https://api.example.com/metrics": context deadline exceeded (Client.Timeout exceeded while reading body)

This is the http input timeout — distinct from [outputs.http] ... context deadline exceeded (writing metrics out) and from an outright connection refused. Here Telegraf reached the host but the endpoint answered too slowly.

Symptoms

• Metrics from a specific [[inputs.http]] URL are missing while other inputs report normally.
• journalctl -u telegraf repeats context deadline exceeded ... awaiting headers on each interval.
• The gather duration for the http input hovers near its timeout value every cycle.
• curl against the same URL succeeds but takes several seconds to return the first byte.
• The failure is intermittent and correlates with the endpoint’s own load or a slow upstream dependency.

Common Root Causes

• timeout too low for a slow endpoint — the default 5s is shorter than the endpoint’s time-to-first-byte under load.
• Slow upstream / database query behind the endpoint — the API computes metrics on demand and blocks until a backend responds.
• DNS resolution latency — a slow or failing resolver eats seconds before the connection even opens, counted against the same deadline.
• Proxy or egress gateway latency — an HTTP_PROXY/corporate egress hop adds delay or queues the request.
• TLS handshake overhead — a distant endpoint or OCSP/CRL check stretches the handshake past the timeout.
• Large response bodies — a big payload that streams slowly trips the deadline while reading the body.
• Endpoint rate limiting or throttling — the server deliberately delays responses to shed load.

Diagnostic Workflow

First reproduce outside Telegraf and measure where the time goes. curl’s timing breakdown shows whether DNS, connect, TLS, or the server’s processing dominates:

curl -o /dev/null -s -w \
  'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' \
  https://api.example.com/metrics

If ttfb is high but connect/tls are small, the server itself is slow — raise timeout. If dns is large, the resolver is the problem. Run only the http input under Telegraf to confirm the fix:

telegraf --config /etc/telegraf/telegraf.conf --test --input-filter http --debug

A tuned http input raises timeout well above the endpoint’s worst observed TTFB and sets a sane collection interval so a slow endpoint cannot stack requests:

[[inputs.http]]
  urls = ["https://api.example.com/metrics"]
  method = "GET"
  timeout = "30s"
  interval = "60s"
  data_format = "json"

  [inputs.http.tags]
    source = "api-metrics"

If a proxy is involved, make it explicit and confirm the environment the service actually sees. Systemd units do not inherit your shell’s variables:

sudo systemctl show telegraf -p Environment
# add proxy settings via a drop-in if required:
#   [Service]
#   Environment=HTTPS_PROXY=http://10.0.0.9:3128
sudo systemctl edit telegraf

For DNS latency, test resolution time directly and consider a local caching resolver:

dig +stats api.example.com | grep 'Query time'

Example Root Cause Analysis

A platform team’s api_metrics measurements went dark every weekday morning, with Telegraf logging context deadline exceeded (Client.Timeout exceeded while awaiting headers). Overnight the same URL collected fine. curl with the timing template showed ttfb=8.9s during the morning window versus 0.3s at night — the endpoint recomputed a report from a reporting database that was under heavy morning ETL load.

The http input still used the default timeout = "5s", so every morning request was cancelled at five seconds while the server needed nearly nine. Raising timeout = "30s" and lengthening interval to 60s let the slow-but-successful responses complete, and metrics returned. The longer-term fix was a cached endpoint on the API side, but the immediate lesson: a context deadline on the http input usually means the endpoint is slow, not down — measure time-to-first-byte before assuming a network fault.

Prevention Best Practices

• Set timeout from the endpoint’s measured worst-case TTFB with headroom, not the default 5s.
• Keep the plugin interval comfortably larger than timeout so a slow response cannot overlap the next collection.
• Prefer endpoints that return cached/precomputed metrics over ones that compute on every scrape.
• Make proxy and DNS settings explicit in the systemd unit environment and verify with systemctl show.
• Run a local caching resolver on busy Telegraf hosts to remove per-request DNS latency.
• Alert on the http input’s gather time trending toward timeout so you catch slowdowns before data goes missing.

Quick Command Reference

# Measure where the time goes (DNS/connect/TLS/TTFB/total)
curl -o /dev/null -s -w 'dns=%{time_namelookup} ttfb=%{time_starttransfer} total=%{time_total}\n' https://api.example.com/metrics

# Run only the http input with debug
telegraf --config /etc/telegraf/telegraf.conf --test --input-filter http --debug

# Inspect the environment systemd gives Telegraf (proxy vars)
sudo systemctl show telegraf -p Environment

# Check DNS query time
dig +stats api.example.com | grep 'Query time'

# Watch http timeouts live
journalctl -u telegraf -f | grep -i 'deadline exceeded'

Related Guides

• Telegraf Error: [outputs.http] context deadline exceeded
• Telegraf Error: [inputs.http] connection refused
• Telegraf Error: input took longer to collect

More fixes in the Telegraf guides.

Conclusion

[inputs.http] ... context deadline exceeded (Client.Timeout exceeded while awaiting headers) means Telegraf reached the host but the endpoint did not return headers before timeout. Reproduce with curl’s timing breakdown to see whether DNS, TLS, or server processing is the bottleneck, then raise timeout (and interval) to cover the endpoint’s real worst-case latency. Make proxy and resolver settings explicit for the service, and prefer cached endpoints so a busy backend can no longer starve your metrics collection.