🚨 Telegraf Error Guide: '[inputs.kafka_consumer] run out of available brokers' — Fix Broker Connectivity

Overview

The kafka_consumer input joins a consumer group and reads messages from Kafka topics. When the Sarama client cannot establish a session with any broker in the list, it gives up and Telegraf logs the exhausted-brokers error:

2026-07-12T12:00:00Z E! [inputs.kafka_consumer] Error in plugin: kafka: client has run out of available brokers to talk to: dial tcp 10.0.0.9:9092: connect: connection refused

A subtler variant appears when the initial broker answers but returns advertised addresses the client cannot reach — the connection then fails on the advertised host, not the one you configured:

E! [inputs.kafka_consumer] Error in plugin: kafka: client has run out of available brokers to talk to: dial tcp kafka-0.internal:9092: i/o timeout

Telegraf keeps other inputs running, but no messages are consumed from Kafka until a broker session succeeds.

Symptoms

• Metrics parsed from Kafka topics are entirely missing while other inputs report normally.
• journalctl -u telegraf repeats run out of available brokers to talk to every collection cycle.
• The error names an address you never configured (an internal hostname), pointing at advertised.listeners.
• nc -z broker 9092 succeeds but Telegraf still fails — a SASL/TLS handshake or advertised-listener problem.
• Errors mention SASL or tls: handshake failure once plaintext connectivity is confirmed.

Common Root Causes

• Wrong or unreachable brokers list — the bootstrap address/port is wrong, or the broker is down, giving connection refused.
• advertised.listeners mismatch — the broker bootstraps on a reachable IP but advertises an internal hostname/IP the Telegraf host cannot resolve or route to, causing i/o timeout on the advertised address.
• Listener/port confusion — connecting to the controller or an SSL listener port (9093) with plaintext settings, or plaintext port with TLS enabled.
• Missing/incorrect SASL — the cluster requires SASL_PLAINTEXT/SASL_SSL but the input has no sasl_username/sasl_password or the wrong sasl_mechanism.
• TLS not configured or untrusted — the listener is TLS-only and the input lacks tls_ca/enable_tls, or presents an untrusted cert.
• Firewall / security group — TCP/9092 (or 9093) is blocked from the Telegraf host.
• Consumer group / topic issues — wrong consumer_group colliding with another app, or a topics name that does not exist (surfaces after connectivity is fixed).

Diagnostic Workflow

First prove basic reachability to the broker port from the Telegraf host, then prove the protocol layer separately:

# TCP reachability to the bootstrap broker
nc -z -v 10.0.0.9 9092

# What does the broker actually advertise? (run from a host with kafka tools)
kafka-broker-api-versions.sh --bootstrap-server 10.0.0.9:9092
kafka-metadata-quorum.sh --bootstrap-server 10.0.0.9:9092 describe --status 2>/dev/null || true

If nc succeeds but Telegraf still fails, the client is being handed unreachable advertised addresses or the handshake is failing. Inspect advertised listeners on the broker:

grep -E 'listeners|advertised.listeners' /etc/kafka/server.properties

A plaintext consumer config lists all brokers so a single dead one does not exhaust the client:

[[inputs.kafka_consumer]]
  brokers = ["10.0.0.9:9092", "10.0.0.10:9092", "10.0.0.11:9092"]
  topics = ["telegraf-metrics"]
  consumer_group = "telegraf_metrics_consumers"
  offset = "oldest"
  data_format = "influx"

For a SASL_SSL cluster (common on managed Kafka), set the mechanism, credentials, and TLS trust:

[[inputs.kafka_consumer]]
  brokers = ["broker1.example.com:9093", "broker2.example.com:9093"]
  topics = ["telegraf-metrics"]
  consumer_group = "telegraf_metrics_consumers"

  enable_tls = true
  tls_ca = "/etc/telegraf/kafka-ca.pem"
  # insecure_skip_verify = false

  sasl_username = "${KAFKA_USER}"
  sasl_password = "${KAFKA_PASSWORD}"
  sasl_mechanism = "SCRAM-SHA-512"   # or "PLAIN" / "SCRAM-SHA-256" per cluster

Run only the kafka_consumer input with debug to watch the session negotiation:

telegraf --config /etc/telegraf/telegraf.conf --test --input-filter kafka_consumer --debug

If the client is handed an internal hostname it cannot resolve, fix advertised.listeners on the brokers (or add the hostname to DNS/hosts) so advertised addresses are routable from Telegraf.

Example Root Cause Analysis

A Telegraf collector on a monitoring subnet logged [inputs.kafka_consumer] ... run out of available brokers to talk to: dial tcp kafka-0.internal:9092: i/o timeout, even though the config pointed at 10.0.0.9:9092 and nc -z 10.0.0.9 9092 succeeded. The give-away was the error naming kafka-0.internal, a hostname that appeared nowhere in the Telegraf config.

The broker bootstrapped fine on its IP but its advertised.listeners returned PLAINTEXT://kafka-0.internal:9092. Sarama connected to the bootstrap IP, fetched metadata, and then tried to reach every partition leader by its advertised name — which did not resolve from the monitoring subnet, producing an i/o timeout. Adding kafka-0.internal (and its peers) to DNS made the advertised addresses routable and consumption started immediately. The lesson: when the exhausted-brokers error names an address you never configured, it is advertised.listeners, not your brokers list — fix what the broker advertises, not just how you connect.

Prevention Best Practices

• List every broker in brokers = [...] so one dead node cannot exhaust the client on startup.
• Ensure advertised.listeners returns addresses that are resolvable and routable from every consumer, including monitoring hosts.
• Connect to the correct listener/port for your security protocol (plaintext vs 9093 TLS); do not mix them.
• Match sasl_mechanism exactly to the cluster (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512) and store secrets in ${ENV_VARS}.
• Provide tls_ca for private CAs rather than setting insecure_skip_verify = true.
• Use a dedicated consumer_group for Telegraf so it does not compete with application consumers for partitions.

Quick Command Reference

# TCP reachability to the bootstrap broker
nc -z -v 10.0.0.9 9092

# Verify protocol and what the broker advertises
kafka-broker-api-versions.sh --bootstrap-server 10.0.0.9:9092
grep -E 'advertised.listeners' /etc/kafka/server.properties

# Run only the kafka_consumer input with debug
telegraf --config /etc/telegraf/telegraf.conf --test --input-filter kafka_consumer --debug

# Tail broker connection errors live
journalctl -u telegraf -f | grep -i kafka

Related Guides

• Telegraf kafka circuit breaker is open
• Telegraf HTTP connection refused
• Telegraf x509 certificate signed by unknown authority

More fixes in the Telegraf guides.

Conclusion

[inputs.kafka_consumer] ... client has run out of available brokers to talk to: ... connection refused means the Sarama client could reach no broker to form a session — usually a wrong brokers list, an advertised.listeners mismatch, or a SASL/TLS handshake failure. Prove port reachability with nc, check what the broker advertises, and align brokers, sasl_mechanism, and TLS trust in the config. When the error names an address you never configured, fix advertised.listeners, and Telegraf will consume Kafka topics reliably.