🚨 Telegraf Error Guide: 'Error writing to outputs.influxdb: could not write' — Fix Connectivity

Overview

The agent logs this whenever the InfluxDB output plugin cannot complete an HTTP write to any of its configured URLs. The wrapped Go network error tells you which layer failed:

E! [agent] Error writing to outputs.influxdb: could not write any address: Post "http://influxdb:8086/write?db=telemetry": dial tcp 10.0.3.12:8086: connect: connection refused

Common variants of the same failure:

E! [agent] Error writing to outputs.influxdb: could not write any address: ... dial tcp: lookup influxdb: no such host
E! [agent] Error writing to outputs.influxdb: could not write any address: ... context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Unlike a 4xx, these are retryable transport failures — Telegraf buffers and retries, so metrics survive until the buffer overflows.

Symptoms

• Error writing to outputs.influxdb: could not write any address repeating each flush.
• The wrapped error is one of connection refused, no such host, i/o timeout, or context deadline exceeded.
• Metrics accumulate, then metric buffer overflow warnings begin.
• InfluxDB is up for other clients, or is genuinely down/unreachable from this host.

Common Root Causes

• InfluxDB is down or restarting — nothing is listening on the port (connection refused).
• DNS resolution failure — the hostname does not resolve from Telegraf’s host (no such host).
• Network path blocked — firewall, security group, or NetworkPolicy dropping the connection (i/o timeout).
• Wrong port or scheme — http to a TLS-only port, or the wrong port number.
• InfluxDB overloaded — accepting connections but too slow to answer within timeout (context deadline exceeded).
• Container/pod networking — service name not reachable from Telegraf’s network namespace.

Diagnostic Workflow

Read the wrapped error first; it dictates everything else:

journalctl -u telegraf --since '10 min ago' | grep 'could not write' | tail -5

Resolve the hostname from Telegraf’s own host to rule out DNS:

getent hosts influxdb
nslookup influxdb

Test raw TCP reachability to the port, independent of HTTP:

nc -vz influxdb 8086
timeout 5 bash -c 'cat < /dev/null > /dev/tcp/influxdb/8086' && echo "port open"

Reproduce the exact write and time it so you can tell “refused” from “slow”:

curl -i -m 5 -XPOST "http://influxdb:8086/write?db=telemetry" \
  --data-binary 'probe,host=diag value=1' \
  -w '\nhttp=%{http_code} time=%{time_total}s\n'

Confirm the plugin URL and timeout match reality:

[[outputs.influxdb]]
  urls = ["http://influxdb:8086"]
  timeout = "5s"

connection refused → InfluxDB not listening (check the service). no such host → DNS. timeout/deadline exceeded → firewall or an overloaded server; raise timeout only if the server is genuinely slow, not to mask a blocked path.

Example Root Cause Analysis

After a Kubernetes node drain, Telegraf on one node began logging could not write any address: dial tcp: i/o timeout while agents on other nodes were fine. DNS resolved correctly (getent hosts returned the ClusterIP), but nc -vz influxdb 8086 hung — a newly-applied NetworkPolicy allowed InfluxDB ingress only from pods labeled role=collector, and the rescheduled Telegraf pod had lost that label after a template edit. Restoring the role=collector label re-permitted the path; nc connected instantly and the buffered metrics flushed on the next interval. The i/o timeout (not refused) was the tell that the packet was being dropped, not rejected.

Prevention Best Practices

• Configure multiple urls (or an HA load balancer) so a single InfluxDB node failing does not stop writes.
• Set a sane timeout (3–10s) and size metric_buffer_limit to survive expected restart windows so nothing drops during brief outages.
• Alert on the output error rate; a rising could not write count is your earliest signal of a broken path.
• Monitor DNS and NetworkPolicy/security-group changes that can silently sever the Telegraf→InfluxDB path.
• Add disk buffering for outages longer than memory can hold, so a prolonged InfluxDB outage does not lose data.

Quick Command Reference

# Read the wrapped network error
journalctl -u telegraf --since '10 min ago' | grep 'could not write'

# DNS + raw TCP reachability
getent hosts influxdb
nc -vz influxdb 8086

# Reproduce and time the write
curl -i -m 5 -XPOST "http://influxdb:8086/write?db=telemetry" \
  --data-binary 'probe value=1' -w '\nhttp=%{http_code} time=%{time_total}s\n'

# Confirm InfluxDB is listening (on the server)
ss -tlnp | grep 8086

# Watch for the error live
journalctl -u telegraf -f | grep 'could not write'

Conclusion

Error writing to outputs.influxdb: could not write is a transport failure, and the wrapped Go error names the layer: connection refused (nothing listening), no such host (DNS), or i/o timeout / context deadline exceeded (blocked path or overloaded server). Resolve DNS, test the raw port with nc, and reproduce with a timed curl before changing config. Because these errors are retryable, buffer sizing and multiple URLs keep data safe while you restore the path. If the buffer fills before you recover, see the metric buffer overflow guide. More fixes in the Telegraf guides.