🚨 Telegraf Error Guide: 'context deadline exceeded' — Fix Output Write Timeouts

Overview

Telegraf gives every output write a deadline via its timeout setting. When the destination doesn’t respond before that deadline, the underlying Go HTTP client aborts the request and Telegraf logs a timeout error:

2026-07-11T12:00:00Z E! [outputs.influxdb_v2] when writing to [http://influxdb:8086]: Post "http://influxdb:8086/api/v2/write": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Unlike connection refused, the TCP connection usually succeeds — the destination is reachable but too slow (or stalled) to answer within timeout. The batch is not confirmed written, stays buffered, and is retried; if this persists the buffer fills and metrics are dropped.

Symptoms

• Recurring context deadline exceeded or Client.Timeout exceeded while awaiting headers errors on an output.
• Writes succeed intermittently but time out under load or at peak.
• Rising write_time_ns and buffer pressure in the internal plugin, sometimes followed by Metric buffer overflow.
• The destination host is up and TCP-reachable (unlike a refused connection), just slow.
• A curl to the write endpoint is slow or hangs, not instantly rejected.

Common Root Causes

• timeout set too low for the real write latency of the destination.
• Destination overloaded — the store is CPU/IO-saturated (often from high-cardinality ingest) and slow to acknowledge writes.
• Batches too large — a big metric_batch_size produces a request the destination can’t process before the deadline.
• Network latency / packet loss over a WAN or through a proxy/load balancer.
• A proxy or gateway (nginx, Caddy, a cloud LB) buffering or stalling the request before it reaches the store.
• TLS handshake or DNS delays adding seconds to each request.

How to diagnose

Confirm the destination is reachable but slow, and measure the actual latency:

curl -o /dev/null -s -w 'http=%{http_code} connect=%{time_connect}s total=%{time_total}s\n' \
  -XPOST "http://influxdb:8086/api/v2/write?org=ops&bucket=telemetry" \
  -H "Authorization: Token ${INFLUX_TOKEN}" --data-binary 'probe value=1'

A fast connect but large total confirms the endpoint answers TCP quickly but is slow to process the write — a timeout, not a connectivity, problem.

Read the output’s write timing and buffer pressure from the internal plugin:

telegraf --config /etc/telegraf/telegraf.conf --test --input-filter internal \
  | grep -E 'write_time_ns|buffer_size|buffer_limit|metrics_dropped'

Watch the errors live and see whether they cluster at peak load:

journalctl -u telegraf -f | grep -iE 'context deadline|Client.Timeout|write_time'

Fixes

1. Raise timeout to cover real write latency — but keep it under flush_interval. If curl shows ~4s writes, a 2s timeout will always fail:

[agent]
  flush_interval = "10s"

[[outputs.influxdb_v2]]
  urls = ["http://influxdb:8086"]
  token = "${INFLUX_TOKEN}"
  organization = "ops"
  bucket = "telemetry"
  timeout = "8s"          # cover real latency, still < flush_interval

2. Shrink batches so each request completes faster:

[agent]
  metric_batch_size = 500   # smaller requests finish within the deadline

3. Reduce cardinality to speed up the destination — high-cardinality writes are the most common cause of slow ingest:

[[processors.tag_limit]]
  limit = 8
  keep = ["host", "region", "service"]

[[processors.dedup]]
  dedup_interval = "60s"

4. If a proxy sits in front, raise its upstream/read timeouts and disable request buffering so it doesn’t stall the write, or point Telegraf directly at the store.

5. Fix TLS/DNS overhead — pin an IP or fix resolution, and reuse connections; persistent slow handshakes add to every request’s deadline.

6. If the destination is simply undersized, scale it or front it with a durable queue output so Telegraf isn’t blocked on synchronous, slow writes.

What to watch out for

• Raising timeout above flush_interval lets a stalled write freeze the whole flush cycle — keep timeout < flush_interval.
• A timeout means the write was not confirmed; if the destination actually processed it, a retry can duplicate data — prefer idempotent writes (line protocol with explicit timestamps) so retries are safe.
• Don’t just enlarge the buffer; that delays drops but doesn’t fix slow writes.
• Alert on internal_write.write_time_ns trending up — it precedes timeouts and overflow.
• Test after tuning by watching for zero metrics_dropped through a real peak, not just a quiet period.

Related

• error writing to outputs — could not write
• connection refused to an output
• metric buffer overflow — dropped metrics on a full buffer

More fixes in the Telegraf guides.