🚨 Filebeat Error: 'kafka: client has run out of available brokers to talk to' — Cause, Fix, and Troubleshooting Guide

Overview

Filebeat’s kafka output bootstraps by connecting to the brokers you list, then discovers the full broker set from metadata. When it cannot reach any broker — or the addresses it learns are unreachable — the underlying client gives up:

Failed to connect to broker kafka01:9092: dial tcp: lookup kafka01 on 10.0.0.2:53: no such host
kafka: client has run out of available brokers to talk to: EOF

The tell is that Filebeat exhausted every broker it knows and none accepted a connection. The usual culprits are wrong bootstrap addresses, brokers advertising internal hostnames the Filebeat host cannot resolve, or a TLS/SASL handshake that fails so every broker is marked unavailable. No events publish until at least one broker is reachable and the handshake succeeds.

Symptoms

• kafka: client has run out of available brokers to talk to (often with EOF or dial tcp before it).
• Failed to connect to broker <host>:<port> for each configured broker.
• Works from the broker host but not from the Filebeat host (advertised-listener mismatch).
• Started after enabling TLS/SASL or changing advertised.listeners on Kafka.

Common Root Causes

• Wrong bootstrap hosts — unreachable address or port.
• Advertised listeners returning internal hostnames Filebeat cannot resolve.
• TLS handshake failure — untrusted CA, hostname mismatch, or plaintext-vs-TLS mismatch.
• SASL auth failure — wrong mechanism/credentials, marking brokers unavailable.
• Firewall/security group blocking 9092/9093 from the Filebeat host.
• Kafka down or the topic’s leader broker offline.

How to diagnose

Confirm basic reachability from the Filebeat host:

nc -vz kafka01 9092
getent hosts kafka01

Run Filebeat with output debug to see the metadata/broker negotiation:

filebeat -e -d "kafka" -c /etc/filebeat/filebeat.yml

Check what the brokers advertise — a mismatch here is the classic cause:

kafka-broker-api-versions.sh --bootstrap-server kafka01:9092 2>&1 | head
# or inspect server.properties: advertised.listeners=

Fixes

List reachable brokers and match the advertised-listener names your host can resolve:

output.kafka:
  hosts: ["kafka01:9092", "kafka02:9092", "kafka03:9092"]
  topic: "filebeat-logs"
  partition.round_robin:
    reachable_only: true
  required_acks: 1

If the brokers advertise internal names, add host entries or fix advertised.listeners on Kafka so the returned addresses resolve from the Filebeat host. For TLS:

output.kafka:
  hosts: ["kafka01:9093"]
  ssl.certificate_authorities: ["/etc/filebeat/certs/kafka-ca.pem"]
  ssl.verification_mode: "full"

For SASL, set the mechanism and credentials the cluster expects:

output.kafka:
  username: "filebeat"
  password: "${KAFKA_PASS}"
  sasl.mechanism: "SCRAM-SHA-512"

Test the output after each change:

filebeat test output

What to watch out for

• The bootstrap hosts only start discovery; the advertised listeners must also be reachable.
• A plaintext client against a TLS listener (or vice versa) fails every broker with cryptic EOFs.
• reachable_only: true avoids stalling on a dead partition leader.
• SASL mechanism must match the broker exactly (PLAIN vs SCRAM-SHA-256/512).

Related

• Filebeat Error Guide: ‘connection refused’
• Filebeat Error Guide: ‘backoff(elasticsearch(…)): lookup: no such host’
• Filebeat Error: ‘Connection marked as failed because the onConnect callback failed’