🚨 Telegraf Error Guide: 'outputs.prometheus_client bind: address already in use' — Fix Port

Overview

The Prometheus client output starts an HTTP listener that Prometheus scrapes. If the chosen port is already bound, Telegraf fails to start the output:

E! [outputs.prometheus_client] error listening: listen tcp :9273: bind: address already in use

Depending on version this can be fatal to startup or leave Telegraf running with a dead scrape endpoint. Either way, Prometheus stops getting metrics from this instance.

Symptoms

• listen tcp :9273: bind: address already in use at startup or on config reload.
• Prometheus targets page shows this instance as down with a connection refused on /metrics.
• systemctl restart telegraf sometimes works, sometimes not (race with the old process releasing the port).
• Two Telegraf units, or Telegraf plus another exporter, are configured on the same port.

Common Root Causes

• A previous Telegraf process still holds the port — an old instance did not exit, or a reload spawned a second listener.
• Another exporter uses the same port — 9273 collides with a custom exporter, or two Telegraf configs both pick 9273.
• TIME_WAIT sockets briefly holding the port after a fast restart.
• A second [[outputs.prometheus_client]] block in an included config file under /etc/telegraf/telegraf.d/.
• Host networking in a container where the port is already taken on the node.

Diagnostic Workflow

Identify exactly what owns the port:

sudo ss -tlnp | grep 9273
sudo lsof -iTCP:9273 -sTCP:LISTEN

LISTEN 0 4096 *:9273 *:* users:(("telegraf",pid=8123,fd=9))

Check for duplicate Telegraf processes or a second listen block:

pgrep -a telegraf
grep -rn 'prometheus_client' /etc/telegraf/telegraf.conf /etc/telegraf/telegraf.d/

Inspect the effective listen address in config:

[[outputs.prometheus_client]]
  listen = ":9273"
  path = "/metrics"

If a stale Telegraf holds the port, stop the service cleanly and confirm the port frees:

sudo systemctl stop telegraf
sleep 2
sudo ss -tlnp | grep 9273 || echo "9273 is free"
sudo systemctl start telegraf

Once bound correctly, verify the endpoint actually serves metrics:

curl -s http://localhost:9273/metrics | head

Example Root Cause Analysis

A config-management change added a drop-in at /etc/telegraf/telegraf.d/prometheus.conf while the main telegraf.conf still had its own [[outputs.prometheus_client]] on :9273. Telegraf loads and merges every file in telegraf.d, so two listeners tried to bind the same port; the second failed with address already in use. grep -rn prometheus_client across both paths revealed the duplicate. Removing the block from the main config (keeping only the managed drop-in) left a single listener, and curl localhost:9273/metrics returned data. To prevent recurrence they added a CI check rejecting more than one prometheus_client block across all Telegraf config files.

Prevention Best Practices

• Keep exactly one [[outputs.prometheus_client]] across telegraf.conf and every file in telegraf.d/; enforce it in CI.
• Assign each exporter/agent a unique, documented port and record 9273 as owned by Telegraf.
• Use systemctl stop then verify the port is free before start when scripting restarts, to avoid the release race.
• In containers, prefer explicit port mappings over host networking so collisions surface at deploy time.
• Alert on the Prometheus target being down so a failed bind is caught immediately, not at next incident.

Quick Command Reference

# Find what owns the port
sudo ss -tlnp | grep 9273
sudo lsof -iTCP:9273 -sTCP:LISTEN

# Find duplicate processes / config blocks
pgrep -a telegraf
grep -rn 'prometheus_client' /etc/telegraf/telegraf.conf /etc/telegraf/telegraf.d/

# Clean restart that frees the port
sudo systemctl stop telegraf && sleep 2 && sudo systemctl start telegraf

# Verify the endpoint serves metrics
curl -s http://localhost:9273/metrics | head

Conclusion

outputs.prometheus_client bind: address already in use means something already holds the listen port — usually a stale Telegraf process or a duplicate prometheus_client block merged in from telegraf.d/. Use ss/lsof to find the owner and grep -rn to find duplicate config, then keep a single listener on a uniquely-assigned port. Verify the fix with a curl to /metrics. For write-side failures to a push destination, see the could not write guide. More fixes in the Telegraf guides.