🚨 Telegraf Error Guide: '[outputs.graphite] connection refused' — Fix Carbon Plaintext Delivery

Overview

The graphite output sends metrics to Carbon over the plaintext line protocol, TCP port 2003 by default. When nothing is listening on that host and port, the TCP connect is refused and the plugin logs that it could not write to any server in the cluster:

2026-07-12T12:00:00Z E! [outputs.graphite] Error in plugin: could not write to any Graphite server in cluster: dial tcp 10.0.0.8:2003: connect: connection refused

If the host is unreachable rather than refusing, the same plugin instead reports a timeout:

E! [outputs.graphite] Error in plugin: could not write to any Graphite server in cluster: dial tcp 10.0.0.8:2003: i/o timeout

Until a server accepts the connection, metrics buffer in Telegraf and are eventually dropped once the metric buffer fills.

Symptoms

• No new metrics appear in Graphite/Grafana while Telegraf keeps collecting.
• journalctl -u telegraf repeats could not write to any Graphite server in cluster: ... connection refused on each flush.
• nc -z 10.0.0.8 2003 fails from the Telegraf host but Carbon appears “running” on the server.
• The error started after Carbon was reconfigured, moved ports, or switched from plaintext (2003) to the pickle port (2004).
• Telegraf later logs metric buffer overflow because the failed output backs up the buffer.

Common Root Causes

• Carbon plaintext listener not running — carbon-cache/go-carbon is down or the plaintext receiver is disabled, so 2003 is closed.
• Wrong port — the config points at 2003 but Carbon’s plaintext receiver listens elsewhere, or the config accidentally targets the pickle port 2004.
• TCP vs UDP mismatch — Telegraf’s graphite output is TCP by default; a UDP-only Carbon receiver refuses/ignores the TCP connect.
• Firewall or security group blocking 2003 — a host firewall or cloud security group drops the connection between Telegraf and Carbon.
• Carbon bound to loopback — the receiver listens on 127.0.0.1:2003 only, so remote Telegraf hosts are refused.
• Wrong host/DNS — the servers entry resolves to the wrong or a decommissioned host.

Diagnostic Workflow

First confirm the plaintext port is actually open from the Telegraf host — this separates a network/listener problem from a Telegraf one:

nc -z -v 10.0.0.8 2003
# Send a test metric by hand; it should appear in Graphite
echo "test.telegraf.probe 1 $(date +%s)" | nc -q1 10.0.0.8 2003

On the Carbon server, verify what is listening on 2003:

sudo ss -ltnp | grep ':2003'

Run only the graphite output under test to reproduce the write path:

telegraf --config /etc/telegraf/telegraf.conf --test --debug

The graphite output config lists one or more Carbon servers and a metric prefix/template:

[[outputs.graphite]]
  servers = ["10.0.0.8:2003"]
  prefix = "telegraf"
  template = "host.tags.measurement.field"
  graphite_tag_support = false
  timeout = 5

If your Carbon receiver only speaks UDP, tell the plugin to use UDP instead of the default TCP:

[[outputs.graphite]]
  servers = ["10.0.0.8:2003"]
  prefix = "telegraf"
  # graphite_udp is set on the output via the connection; for UDP use:
  # (Carbon must have a UDP_RECEIVER_ENABLED listener on 2003)

For a Carbon bound to loopback, rebind it to a reachable interface (e.g. LINE_RECEIVER_INTERFACE = 0.0.0.0 in carbon.conf) and restart Carbon, then re-test the nc probe.

Example Root Cause Analysis

After a Graphite server migration, every Telegraf agent began logging could not write to any Graphite server in cluster: dial tcp 10.0.0.8:2003: connect: connection refused, and dashboards stopped updating. nc -z 10.0.0.8 2003 failed from the agents, yet the ops team insisted Carbon was running — systemctl status go-carbon was green.

Running ss -ltnp | grep 2003 on the server revealed the receiver was bound to 127.0.0.1:2003: the new carbon.conf shipped with LINE_RECEIVER_INTERFACE = 127.0.0.1, so Carbon only accepted local connections and refused every remote agent. Changing the interface to 0.0.0.0, restarting Carbon, and confirming with the manual echo ... | nc probe restored delivery across the fleet. The lesson: “connection refused” means something answered the address and rejected the connection — check what the listener is actually bound to, not just whether the service is running.

Prevention Best Practices

• Verify Carbon’s listener binds to a reachable interface (0.0.0.0 or the management IP), not 127.0.0.1, after any migration.
• Keep the plaintext port explicit and consistent (2003 for line protocol; 2004 is pickle and is not what the graphite output speaks by default).
• Add a synthetic echo "..." | nc <host> 2003 probe to monitoring so a closed receiver is caught before dashboards go stale.
• List multiple servers for redundancy so a single Carbon node failure does not stop delivery.
• Open port 2003 explicitly in host firewalls and cloud security groups between Telegraf and Carbon.
• Watch Telegraf’s metric buffer overflow warnings — they are the downstream symptom of a failing graphite output.

Quick Command Reference

# Confirm the plaintext port is open from the Telegraf host
nc -z -v 10.0.0.8 2003

# Send a manual test metric to Carbon
echo "test.telegraf.probe 1 $(date +%s)" | nc -q1 10.0.0.8 2003

# On the Carbon server: what is bound to 2003?
sudo ss -ltnp | grep ':2003'

# Run the outputs under test
telegraf --config /etc/telegraf/telegraf.conf --test --debug

# Watch graphite write errors live
journalctl -u telegraf -f | grep -i graphite

Related Guides

• Telegraf HTTP connection refused
• Telegraf output context deadline exceeded
• Telegraf metric buffer overflow

More fixes in the Telegraf guides.

Conclusion

[outputs.graphite] could not write to any Graphite server in cluster: ... connection refused means no Carbon receiver accepted the TCP connection on port 2003 — usually because the listener is down, bound to loopback, on the wrong port, or blocked by a firewall. Confirm with nc -z and a manual echo ... | nc probe, check what Carbon is bound to with ss -ltnp, and align the port and TCP/UDP mode. Fix the listener and metrics flow into Graphite again before the buffer overflows.