Kafka Broker, KRaft & Cluster Errors
The failures that take a Kafka broker or the whole cluster down: a broker that won't start, a controller that isn't available, an inconsistent cluster ID after a bad restart, offline partitions, corrupted log segments, or KRaft/ZooKeeper metadata problems. Each guide opens with the direct cause and the `kafka-*.sh`, `kafka-metadata-quorum`, and log commands to confirm it. Start with your symptom below, or paste the broker log into the assistant.
Fix your error now
Paste the error or logs and let the Incident Assistant diagnose it, or run a guided triage prompt with AI — no copy-paste.
Every guide in this cluster
50 guides-
'Fatal error during KafkaServer startup' Broker Won't Start
Fix Kafka 'Fatal error during KafkaServer startup. Prepare to shutdown': resolve bad config, port-in-use, log.dir failures, and meta.properties cluster.id/broker.id mismatches.
-
'Controller not available' No Active Controller
Fix Kafka 'Controller not available / controller connection failed': diagnose quorum loss, no elected controller, ZooKeeper outages, and KRaft voter majority failures.
-
'The Cluster ID doesn't match stored clusterId' — Fixing InconsistentClusterIdException
Fix Kafka's InconsistentClusterIdException when a broker's meta.properties clusterId no longer matches the cluster: diagnose stale data dirs, wrong KRaft storage format, and orphaned brokers, then recover safely.
-
'Partition marked offline' OfflinePartitionsCount > 0
Diagnose Kafka offline partitions when OfflinePartitionsCount is above zero and a partition has no leader. Restore replicas and recover offline log dirs.
-
'[KafkaServer id=1] shutting down' Graceful vs Crash
Read Kafka '[KafkaServer id=1] shutting down' and 'started' lifecycle lines: tell a graceful controlled.shutdown from an abnormal crash and trace the real trigger.
-
'Broker may not be available' Connection Failure
Fix Kafka 'Connection to node 1 could not be established. Broker may not be available': diagnose down brokers, wrong bootstrap servers, listeners, and firewalls.
-
'BrokerEndPointNotAvailableException' Missing Listener
Fix Kafka BrokerEndPointNotAvailableException: a listener or security protocol has no advertised endpoint. Fix listeners, advertised.listeners, and listener maps.
-
'ConnectionLoss for /brokers/ids' ZooKeeper Connection Loss
Fix Kafka ZooKeeper ConnectionLoss for /brokers/ids: diagnose a downed ensemble, lost quorum, port 2181 firewall blocks, bad zookeeper.connect, and GC pauses.
-
'Controller epoch is older than the current controller epoch' Stale Epoch
Fix Kafka 'controller epoch is older than the current controller epoch': understand epoch fencing, split brain after a network partition, and how to confirm the live controller.
-
'Controller heartbeat timeout' Broker Fenced in KRaft
Fix Kafka KRaft 'controller heartbeat timeout / broker fenced': tune broker.heartbeat.interval.ms and broker.session.timeout.ms, and diagnose missed broker heartbeats.
-
'Controller mutation rate quota exceeded' Throttled Topic Ops
Fix Kafka controller mutation rate quota errors: understand CONTROLLER_MUTATION quotas, throttled topic create/delete/partition ops, and how to size the limit safely.
-
'Corrupted index found' Offset Index Corrupted on Startup
Fix Kafka 'Corrupted index found' and 'Found invalid offset index' errors: understand index rebuilds on restart, time index corruption, and slow recovery startups.
-
'CorruptRecordException' Message Failed Its CRC Checksum
Fix Kafka CorruptRecordException: diagnose CRC32C checksum mismatches from network corruption, bad disks, truncated segments after unclean shutdown, and consumer fetch detection.
-
'Could not recover log' Recovery Failure After Crash
Fix Kafka 'Could not recover log' errors: diagnose crash recovery failures, 'Unable to allocate log segment', disk-full recovery, and brokers stuck on startup.
-
'Error for partition topic-0 at offset 12345' ReplicaFetcherThread Failure
Decode ReplicaFetcherThread errors when a Kafka follower can't fetch from the leader: NOT_LEADER, OFFSET_OUT_OF_RANGE, fetch size, and TLS causes.
-
'Error in log cleaner thread' Retention Cleanup Failed
Fix Kafka log cleaner and retention cleanup failures: diagnose a dead LogCleaner thread, dedupe buffer memory limits, compaction errors, and growing disk usage.
-
'Error while creating log directories' Log Dir Failure
Fix Kafka 'Error while creating log directories': resolve missing log.dirs paths, wrong ownership, Permission denied, full disks, and stale .lock files that mark a dir offline.
-
'Error while electing or becoming controller on broker 1' Election Failure
Fix Kafka 'Error while electing or becoming controller on broker 1': diagnose ZooKeeper session loss, quorum problems, znode conflicts, and stuck controller election.
-
'Error while fetching metadata' LEADER_NOT_AVAILABLE
Fix Kafka's 'Error while fetching metadata ... LEADER_NOT_AVAILABLE' and UNKNOWN_TOPIC_OR_PARTITION client warnings: causes, diagnostics, and resolution.
-
'Error while flushing log' fsync Failure on Broker
Fix Kafka 'Error while flushing log for topic-0' fsync failures: diagnose disk stalls, IO errors, and storage latency that mark a log directory offline via KafkaStorageException.
-
'Failed to append metadata record' Raft Append Failure
Fix KRaft 'Failed to append metadata record' to __cluster_metadata: diagnose lost leadership, no quorum, disk-full, and timeout failures on the Raft write path.
-
'Failed to append records to topic-0 in dir /var/lib/kafka/data' Offline Log Dir
Fix Kafka's KafkaStorageException when a broker fails to append to its local log and marks the data directory offline due to disk, IO, or permission faults.
-
'Failed to update metadata after 60000 ms' Client Timeout
Fix Kafka 'TimeoutException: Failed to update metadata after 60000 ms': resolve bad bootstrap.servers, broken advertised.listeners, ACL denials, and unreachable brokers.
-
'Found a corrupted segment' Corrupt Log Segment on Load
Fix Kafka corrupted log segment errors: diagnose unclean shutdowns, truncated segments, and 'Unexpected EOF while reading log' so a broker can finish startup recovery.
-
'InvalidPartitionsException' Cannot Decrease Partition Count
Fix Kafka InvalidPartitionsException when altering a topic: why partitions can only increase, invalid counts, IaC drift, and the ordering caveat for keyed messages.
-
'InvalidReplicationFactorException' Larger Than Available Brokers
Fix Kafka InvalidReplicationFactorException: replication factor larger than available brokers, brokers down, single-node dev RF=3 defaults, and min.insync.replicas confusion.
-
'KafkaStorageException: Disk error when trying to access log file' — Recover the Offline Log Directory
Fix KafkaStorageException disk errors in Kafka: diagnose offline log directories, failed disks, and space problems, then recover partitions safely.
-
'Leader election failed' Offline Partitions and No Leader
Why Kafka controller and preferred-leader elections fail, how unclean.leader.election leaves partitions leaderless, and read-only commands to diagnose it.
-
'Leader epoch mismatch' Fencing in the KRaft Metadata Log
Fix KRaft 'Leader epoch mismatch': understand epoch fencing after a controller election, diagnose stale leaders and divergent followers, and recover the quorum cleanly.
-
'Metadata loader failed' Broker Cannot Apply Controller Updates
Fix KRaft 'Metadata loader failed': diagnose why a broker cannot apply __cluster_metadata updates from the controller due to bad records, version skew, or local faults.
-
'Metadata log corruption detected' Corrupted __cluster_metadata Segment
Fix KRaft 'Metadata log corruption detected': diagnose a corrupted __cluster_metadata segment from CRC mismatch, partial writes, or disk faults, and recover safely.
-
'Metadata quorum unavailable' Controller Majority Down
Fix KRaft 'Metadata quorum unavailable' / 'Quorum controller unavailable': diagnose a lost controller majority, wrong bootstrap controllers, and stalled metadata.
-
'NoBrokersAvailable' Client Cannot Reach Cluster
Fix kafka-python NoBrokersAvailable: diagnose wrong bootstrap_servers, DNS failures, firewall blocks, security protocol mismatches, and down brokers.
-
'Node 1 disconnected' Established Broker Connection Dropped
Fix Kafka 'Node 1 disconnected' and 'Connection to node 1 disconnected' — diagnose idle timeouts, broker restarts, and version or protocol mismatches.
-
'NodeExistsException for /brokers/ids/1' Broker Registration Conflict
Fix Kafka KeeperException NodeExistsException for /brokers/ids: resolve duplicate broker.id, stale ephemeral nodes, session-timeout races, and cloned VM images.
-
'NoNodeException for /brokers/ids/1' Missing ZooKeeper Znode
Fix Kafka KeeperException NoNodeException for /brokers/ids: diagnose wrong chroot in zookeeper.connect, wrong ensemble, fresh-cluster znodes, and tool mismatches.
-
'NOT_LEADER_OR_FOLLOWER' stale partition metadata on clients
Fix Kafka clients hitting NOT_LEADER_OR_FOLLOWER after a leader moves. Understand metadata refresh, retries, advertised.listeners, and why it self-heals.
-
'NotEnoughReplicasException: Messages are rejected' Fewer In-Sync Replicas Than Required
Resolve Kafka NotEnoughReplicasException and NotEnoughReplicasAfterAppendException: ISR dropped below min.insync.replicas under acks=all. Diagnose ISR and fix durability.
-
'Partition reassignment for topic-0 failed' Stuck and Failed Reassignments
Why kafka-reassign-partitions.sh reports a reassignment as still in progress or failed, how to diagnose throttles, dead brokers, and disk, and how to recover.
-
'Raft leader election failed' No Quorum Leader Elected
Fix KRaft 'Raft leader election failed': diagnose missing quorum leader, bad controller.quorum.voters, network partitions, and clock/epoch issues between controllers.
-
'ReplicaNotAvailableException' Replica Reassignment Notice
Understand Kafka ReplicaNotAvailableException: usually transient and informational during reassignment, when to ignore it, and when a replica is truly offline.
-
'Session expired for /controller' ZooKeeper Session Expiry
Fix Kafka ZooKeeper SessionExpiredException for /controller: diagnose long GC pauses, low session timeouts, lost ephemeral nodes, controller re-election, and clock skew.
-
'Shrinking ISR from 1,2,3 to 1,2' Replica Lag Flapping
Why Kafka logs 'Shrinking ISR' and 'Expanding ISR' for a partition, how replica.lag.time.max.ms drives it, and how to stabilize a flapping follower.
-
'Shrinking ISR' replica lagging and under-replicated partitions
Fix Kafka followers that lag and drop out of ISR causing under-replicated partitions: slow disk, NIC saturation, fetchers, and leftover replication throttles.
-
'Snapshot generation failed' Metadata Snapshot Write Error
Fix KRaft 'Snapshot generation failed': diagnose disk-full, permissions, and I/O errors when the controller writes a __cluster_metadata snapshot to checkpoint state.
-
'Stopping serving logs in dir /var/lib/kafka' Log Directory Failure
Fix Kafka KafkaStorageException log directory failures: diagnose disk errors, full volumes, bad permissions, and offline JBOD log dirs marked dead by the broker.
-
'This is not the correct controller for this cluster' Controller Moved
Fix Kafka 'not the correct controller / controller moved to another broker': understand normal failover, stale controllers, and how to confirm the real active controller.
-
'Truncating partition topic-0 to local high watermark 10042' Replica Divergence
Understand Kafka follower log truncation and high watermark mismatch after a leader change, when it is safe, and when unclean leader election causes data loss.
-
'Unable to fetch metadata log' Follower Far Behind
Fix KRaft 'Unable to fetch metadata log' / 'Unable to catch up to metadata log': diagnose a follower controller or broker lagging the __cluster_metadata leader.
-
'UnknownTopicOrPartitionException' Topic Not Found
Fix Kafka UnknownTopicOrPartitionException 'server does not host this topic-partition': missing topics, auto-create disabled, stale metadata, and typos.
Kafka AI prompts
Copy-paste, production-safe prompts for this stack.
Kafka command center
Top errors, validators, and runbooks for the whole stack.
Frequently asked questions
Why does my Kafka broker fail with "Fatal error during KafkaServer startup"?
meta.properties, a port already in use, or corrupt metadata. Read the stack trace under it and check the log-dir permissions. See KafkaServer startup.What causes "The Cluster ID doesn't match" / inconsistent cluster ID?
meta.properties cluster ID differs from the controller/quorum's — usually after a wiped or restored volume, or mixing KRaft and ZooKeeper state. Never just delete meta.properties on a live cluster; reconcile it deliberately. See inconsistent cluster ID.