Skip to content
DevOps AI ToolKit
Newsletter
Kafka Troubleshooting Toolkit

Kafka Troubleshooting Toolkit

Use this Kafka troubleshooting toolkit to diagnose consumer lag, under-replicated and offline partitions, rebalance storms, broker and controller failures, and KRaft/ZooKeeper quorum loss.

Paste your error and get a triage plan.

Paste a log line, CLI error, stack trace, service failure, or config snippet and get a structured troubleshooting plan. Your snippet is carried straight into the AI Incident Response Assistant with Kafka context prefilled.

Open the Kafka Incident Assistant → Browse Kafka prompts

Do not paste secrets, tokens, private keys, passwords, or customer data. Your snippet stays in your browser until you open the assistant.

Top 25 Kafka Errors and Failure Modes

The production failures engineers hit most — each links to a full cause → fix → prevention guide.

Consumer lag growing

Consumers fall behind producers because processing throughput cannot keep pace with the incoming partition offset rate.

Under-replicated partitions / ISR shrink

Follower replicas drop out of the in-sync replica set when they cannot fetch fast enough from the partition leader.

Offline partitions

A partition has no available leader, so producers and consumers cannot read from or write to it until a broker recovers.

LeaderNotAvailableException

Clients hit a partition whose leader is being elected or has just failed, leaving the leader temporarily unknown.

NotLeaderOrFollowerException

A client sends a request to a broker that is no longer the leader or follower for the target partition after a metadata change.

Broker may not be available

Clients cannot reach a broker that is down, unregistered, or unreachable, causing metadata refresh and request failures.

Replica lagging

A follower replica trails the leader's log end offset and risks being removed from the ISR under sustained load.

Leader election failed / unclean election

Controller leader election stalls or promotes an out-of-sync replica, risking data loss when no in-sync leader is available.

RebalanceInProgressException (rebalance storms)

Consumer group membership churns repeatedly, triggering back-to-back rebalances that stall consumption across all members.

CommitFailedException

An offset commit is rejected because the consumer was evicted from the group after exceeding the max poll interval.

RecordTooLargeException

A produced message exceeds the broker or topic max message size limit and is rejected before it can be appended.

Producer TimeoutException on send

A produce request is not acknowledged within delivery.timeout.ms, often due to broker load, network latency, or ISR issues.

Log directory failure (disk full)

A broker marks a log directory offline after an I/O error or full disk, taking its hosted partitions out of service.

Log retention / cleanup failed

Retention or compaction cannot delete or compact segments, causing disk usage to grow unbounded on the broker.

FencedInstanceIdException

A transactional or static-membership producer or consumer is fenced because another instance claimed the same identity.

OutOfOrderSequenceException

The broker detects a gap in the idempotent producer's sequence numbers, indicating lost or reordered in-flight batches.

NetworkException

A transient network fault interrupts a client-broker connection, forcing a retry or metadata refresh on the request.

Controller moved to another broker

The active controller role failed over to a different broker, briefly disrupting metadata and partition-state operations.

Metadata quorum unavailable (KRaft)

The KRaft controller quorum lost majority, so no metadata updates or leader elections can be committed cluster-wide.

ZooKeeper session expired

A broker's ZooKeeper session times out, causing it to deregister and drop leadership of its partitions until it reconnects.

Coordinator unavailable (__consumer_offsets)

The group or transaction coordinator hosting the offsets partition is unreachable, blocking commits and group joins.

ReplicaFetcherThread failed

A follower's replica fetcher thread dies or errors, halting replication for its partitions and shrinking the ISR.

Fetch request timed out

A consumer or follower fetch request exceeds its timeout, signaling high broker latency or an overloaded partition leader.

NoBrokersAvailable / connection refused

A client cannot connect to any bootstrap broker, usually due to wrong addresses, listeners, or all brokers being down.

SASL authentication failed

A client is rejected during the SASL handshake due to bad credentials, mechanism mismatch, or misconfigured JAAS.

Validate your config before you redeploy

Catch the structural mistakes that cause outages — 100% in your browser, nothing uploaded.

YAML Validator (Connect / Strimzi) Not sure? Start with the Incident Assistant →

Kafka Troubleshooting Hubs

Fix a whole class of Kafka errors at once — each hub gathers the related guides for one failure class.

Kafka Broker, KRaft & Cluster Errors

Fix Kafka broker and cluster failures — KafkaServer startup errors, controller not available, inconsistent cluster ID, partition offline, log/storage and KRaft/ZooKeeper metadata errors — with exact commands.

Open hub

Kafka Producer, Consumer & Client Errors

Fix Kafka client failures — producer timeouts, commit-failed and rebalance-in-progress, offset out of range, serialization and record-too-large errors, coordinator and auth failures — with exact fixes.

Open hub

Best Kafka Prompts

Turn symptoms, logs, and config into a structured plan with prompts tuned for Kafka.

Kafka consumer lag investigation

Diagnose and reduce growing consumer group lag

Use this prompt

Kafka consumer rebalance storm triage

Stop repeated rebalances and stabilize the group

Use this prompt

Kafka partition reassignment plan

Rebalance partitions and recover ISR health

Use this prompt

Kafka topic design and partitioning strategy

Choose partition count and replication factor

Use this prompt

Kafka ZooKeeper to KRaft migration plan

Plan quorum cutover and verify KRaft health

Use this prompt

Kafka producer throughput and latency tuning

Tune batching, acks, and delivery timeouts

Use this prompt

Kafka cluster sizing and capacity planning

Size brokers, disk, and partitions for load

Use this prompt

Kafka exactly-once semantics design

Design idempotent and transactional pipelines

Use this prompt

Kafka TLS, SASL, and ACL security hardening

Fix auth failures and lock down access

Use this prompt

Download the Kafka Troubleshooting Runbook Pack

Checklists for brokers, partitions, and consumers that fall behind or drop out.

  • Consumer lag & rebalance triage
  • Under-replicated / offline partitions
  • Broker & controller health
  • KRaft / ZooKeeper quorum
  • Produce/fetch latency & retention
Download the Kafka Runbook Pack

All Kafka Troubleshooting Guides

Grouped by failure-mode type — each guide covers cause, fix, validation, and prevention.

Authentication & TLS (6)

AuthorizationException CertificateExpiredException ClusterAuthorizationException GroupAuthorizationException PKIX path building failed SaslAuthenticationException: Authentication failed

Networking (10)

Address already in use java.io.IOException: Broken pipe Broker may not be available BrokerEndPointNotAvailableException java.io.IOException: Connection reset by peer java.net.SocketTimeoutException: Connection timed out Connection to node -1 could not be established NetworkException: The server disconnected before a response was received Node 1 disconnected Error while accepting connection

Storage (3)

KafkaStorageException: Disk error when trying to access log file OutOfMemoryError: Java heap space Snapshot generation failed

Deployment & Builds (1)

Metadata loader failed

Performance (10)

Failed to acquire lock on file .lock in log directory Too many open files Failed to allocate memory within the configured max blocking time Debugging Kafka Consumer Lag with AI Controller heartbeat timeout Failed to update metadata after 60000 ms Fetch request timed out Shrinking ISR from 1,2,3 to 1,2 TimeoutException: Expiring N record(s) left group due to expired session timeout

API errors (9)

UnsupportedVersionException: The broker does not support Error while fetching metadata Failed to append metadata record Leader epoch mismatch Metadata log corruption detected Metadata quorum unavailable NotLeaderOrFollowerException NOT_LEADER_OR_FOLLOWER RecordTooLargeException

Configuration (6)

InvalidTxnStateException: The producer attempted a transactional operation in an invalid state InvalidConfigurationException: Unknown topic config name InvalidPartitionsException InvalidRecordException: One or more records have been rejected InvalidReplicationFactorException InvalidTopicException

Other (45)

CoordinatorLoadInProgressException: The coordinator is loading GroupMaxSizeReachedException: Consumer group has reached maximum size NotCoordinatorException: This is not the correct coordinator RecordDeserializationException: Error deserializing key/value for partition ThrottlingQuotaExceededException: The throttling quota has been exceeded The Cluster ID doesn ProducerFencedException AI-Assisted Kafka Troubleshooting Explained Designing Kafka Topics: Partitions and Replication CommitFailedException Controller epoch is older than the current controller epoch This is not the correct controller for this cluster Controller mutation rate quota exceeded Controller not available CoordinatorNotAvailableException Found a corrupted segment CorruptRecordException Error while creating log directories Error while electing or becoming controller on broker 1 Failed to append records to topic-0 in dir /var/lib/kafka/data Error while flushing log Could not recover log Fatal error during KafkaServer startup FencedInstanceIdException Truncating partition topic-0 to local high watermark 10042 IllegalGenerationException [KafkaServer id=1] shutting down Raft leader election failed Leader election failed LeaderNotAvailableException Stopping serving logs in dir /var/lib/kafka Error in log cleaner thread No subject alternative names matching IP address found NoBrokersAvailable NotEnoughReplicasException: Messages are rejected Corrupted index found OffsetOutOfRangeException OutOfOrderSequenceException Partition marked offline Partition reassignment for topic-0 failed RebalanceInProgressException Shrinking ISR Error for partition topic-0 at offset 12345 ReplicaNotAvailableException SerializationException: Error serializing Avro message
Browse the full Kafka category

Kafka troubleshooting FAQ

Why is my Kafka consumer lag growing?
Lag grows when consumers process messages slower than producers write them. Common causes are too few consumers or partitions for the load, slow or blocking message handlers, long GC pauses, and frequent rebalances that pause consumption. Compare each partition's log-end offset to the committed offset to find the hot partitions, then scale out consumers, increase partitions, or optimize the processing path. Watch for max.poll.interval.ms breaches, which evict consumers and reset progress.
What causes under-replicated partitions?
A partition is under-replicated when one or more followers fall out of the in-sync replica (ISR) set. This usually stems from a slow or overloaded follower broker, network saturation between brokers, disk I/O pressure, or a broker restart or GC pause. Check UnderReplicatedPartitions and IsrShrinksPerSec metrics, inspect replica fetcher lag, and address the bottlenecked broker's CPU, disk, or network before ISR shrink threatens availability or risks unclean leader election.
How do I fix a Kafka rebalance storm?
Rebalance storms happen when consumers repeatedly join and leave the group, forcing back-to-back partition reassignments. The usual triggers are exceeding max.poll.interval.ms because processing is too slow, session.timeout.ms tuned too tight relative to heartbeats, or unstable pods restarting. Enable cooperative-sticky assignment to avoid stop-the-world rebalances, tune poll and session timeouts to match real processing time, and use static group membership (group.instance.id) so brief restarts do not trigger full reassignments.
Why is a Kafka partition offline?
A partition goes offline when none of its replicas can serve as leader. This typically follows the loss of the broker that hosted the only in-sync replica, a failed or full log directory, or a controller unable to elect a leader. Check the controller and broker logs, confirm the replicas' brokers are up and their log dirs are healthy, and restore or reassign replicas. If unclean.leader.election.enable is false, the partition stays offline until an in-sync replica returns.
KRaft vs ZooKeeper — how do I check quorum?
In ZooKeeper mode, quorum is a majority of ZooKeeper nodes; verify it with the ruok/stat four-letter commands or zkCli and ensure a leader plus enough followers are up. In KRaft mode, metadata quorum is a majority of controller nodes running the Raft log; check it with kafka-metadata-quorum.sh --describe --status to confirm a current leader, the committed offset, and that followers are caught up. Losing majority in either mode halts metadata updates and leader elections cluster-wide.
How do I stop CommitFailedException and producer timeouts?
CommitFailedException means the consumer was removed from the group before committing, almost always because processing exceeded max.poll.interval.ms; reduce max.poll.records, speed up handlers, or raise the interval. Producer TimeoutException on send means a batch was not acknowledged within delivery.timeout.ms, often due to broker overload, ISR shortage, or network latency. Investigate broker health and under-replicated partitions, and tune batch.size, linger.ms, request.timeout.ms, and retries so delivery timeouts reflect real cluster conditions.