🚨 Logstash Error: 'org.apache.kafka.common.errors.TimeoutException' — Cause, Fix, and Troubleshooting Guide

Overview

The Logstash kafka input and output use the standard Kafka client, which first fetches cluster metadata (broker list, topic partitions) before it can produce or consume. When the client cannot reach a broker or fetch metadata for a topic within its timeout, it throws:

[ERROR][logstash.outputs.kafka][main] KafkaProducer.send() failed: org.apache.kafka.common.errors.TimeoutException: Topic events not present in metadata after 60000 ms.

A closely related variant appears when brokers are reachable for metadata but records cannot be delivered before the delivery deadline — usually a broker that is slow, overloaded, or unreachable for the actual produce request:

org.apache.kafka.common.errors.TimeoutException: Expiring 5 record(s) for events-0:120000 ms has passed since batch creation

Both are transport/metadata failures, not data rejections. Either Logstash cannot talk to the brokers at all, it can reach them but the topic does not exist, or produce requests are timing out under load. Until metadata resolves, the input/output cannot make progress and the pipeline blocks.

Symptoms

• TimeoutException: Topic <name> not present in metadata after 60000 ms at startup or on first send.
• Expiring N record(s) ... ms has passed since batch creation under load or during broker instability.
• No events produced to, or consumed from, Kafka; the pipeline stalls and upstream queues back up.
• With SASL/SSL, the log may also show Connection to node -1 could not be established or authentication failures before the timeout.
• kafka-topics.sh --list from the Logstash host either hangs, errors, or omits the expected topic.

Common Root Causes

• Wrong bootstrap_servers — a typo, wrong port, or an address that does not resolve from the Logstash host.
• Broker down or unreachable — the broker is offline, or a firewall blocks TCP 9092 between Logstash and the brokers.
• SASL/SSL misconfiguration — mismatched security_protocol, sasl_mechanism, missing jaas_path/sasl_jaas_config, or a truststore that does not trust the broker cert.
• Missing topic with auto-create disabled — the topic does not exist and the cluster will not create it on demand, so metadata never resolves.
• Timeouts too low under load — request_timeout_ms/delivery_timeout_ms are too small for a busy or slow cluster, so batches expire.

How to diagnose

Prove basic broker reachability from the Logstash host, using the same address list Logstash uses:

# TCP reachable?
nc -vz broker.internal 9092

# Can the client list topics (validates metadata + auth)?
kafka-topics.sh --bootstrap-server broker.internal:9092 --list

# Is the specific topic present with partitions?
kafka-topics.sh --bootstrap-server broker.internal:9092 --describe --topic events

If those hang or fail with an auth error, the problem is connectivity or SASL/SSL, not Logstash. Inspect the current plugin config for the address and security settings:

output {
  kafka {
    bootstrap_servers => "broker.internal:9092"
    topic_id          => "events"
    codec             => json
  }
}

Watch the log live while restarting to catch whether it is a metadata timeout or a delivery expiry:

sudo tail -f /var/log/logstash/logstash-plain.log | grep -Ei 'kafka|TimeoutException|metadata|Expiring'

For SASL, verify the JAAS/mechanism the broker expects; a mismatch shows up as repeated connection failures before the timeout is logged.

Fixes

Set the correct brokers and, for a secured cluster, the full SASL_SSL configuration with a truststore that trusts the broker certificate:

output {
  kafka {
    bootstrap_servers  => "broker1.internal:9092,broker2.internal:9092,broker3.internal:9092"
    topic_id           => "events"
    codec              => json

    security_protocol  => "SASL_SSL"
    sasl_mechanism     => "SCRAM-SHA-512"
    sasl_jaas_config   => "org.apache.kafka.common.security.scram.ScramLoginModule required username='${KAFKA_USER}' password='${KAFKA_PASSWORD}';"
    ssl_truststore_location => "/etc/logstash/kafka/truststore.jks"
    ssl_truststore_password => "${KAFKA_TRUSTSTORE_PW}"

    # Resilience under load / slow brokers
    acks               => "all"
    retries            => 5
    request_timeout_ms => 30000
    delivery_timeout_ms => 120000
  }
}

If the topic is simply missing and auto-create is off, create it explicitly with an appropriate partition/replication count:

kafka-topics.sh --bootstrap-server broker1.internal:9092 \
  --create --topic events --partitions 6 --replication-factor 3

For the Expiring record(s) variant on a busy cluster, raise retries and the timeouts and pick an acks level that matches your durability needs ("1" for lower latency, "all" for no data loss). Then restart and confirm records flow:

sudo systemctl restart logstash
sudo tail -f /var/log/logstash/logstash-plain.log | grep -Ei 'kafka|TimeoutException|Expiring'

What to watch out for

• bootstrap_servers only needs to reach one live broker to bootstrap, but that broker then hands back the advertised.listeners addresses — make sure those are resolvable from the Logstash host too.
• Kafka client port is 9092 (or your cluster’s listener), not the Zookeeper port; check you are testing the right one.
• A metadata timeout with security enabled is often really an auth/TLS failure — read the lines before the TimeoutException.
• delivery_timeout_ms must be >= request_timeout_ms + linger_ms; setting it too low guarantees Expiring record(s) under any load.
• Enabling broker-side auto-create hides missing-topic problems in dev but is usually disabled in production — create topics explicitly.

Related

• Elasticsearch Unreachable
• Beats input connection reset
• Pipeline blocked by backpressure