Kafka Producer, Consumer & Client Errors
The client-side Kafka failures that break producers and consumer groups: producer timeouts and record-too-large, commit-failed and rebalance-in-progress loops, offset-out-of-range, serialization/deserialization errors, coordinator-unavailable, and SASL/SSL auth failures. Each guide leads with the cause and the client-config and `kafka-consumer-groups` commands to fix it. Pick your error below, or hand the client log to the assistant.
Fix your error now
Paste the error or logs and let the Incident Assistant diagnose it, or run a guided triage prompt with AI — no copy-paste.
Every guide in this cluster
45 guides-
'TimeoutException: Expiring N record(s)' Producer Send Timeout
Fix Kafka producer 'TimeoutException: Expiring 5 record(s) ... ms has passed since batch creation': tune delivery.timeout.ms, request.timeout.ms, linger.ms, batch.size and buffer.memory.
-
'CommitFailedException' Offset Commit Cannot Be Completed
Fix Kafka CommitFailedException when the consumer falls out of an active group: diagnose slow processing, max.poll.interval.ms, and rebalance-driven commit rejection.
-
'OffsetOutOfRangeException' Fetch Position Out of Range
Fix Kafka OffsetOutOfRangeException: diagnose offsets behind the log start from retention, auto.offset.reset behavior, and lagging consumers reading deleted data.
-
'RebalanceInProgressException' Consumer Group Is Rebalancing
Fix Kafka RebalanceInProgressException: why offset commits fail mid-rebalance, how cooperative rebalancing changes it, and how to retry the poll cycle safely.
-
'AuthorizationException' Not Authorized to Access
Fix Kafka AuthorizationException: diagnose missing ACLs, wrong principal mapping, allow.everyone.if.no.acl.found, super.users, and authorizer misconfiguration.
-
'CertificateExpiredException' Certificate Has Expired
Fix Kafka CertificateExpiredException: diagnose expired broker or client certs, expired CA roots, clock skew, and short-lived certificate rotation failures.
-
'ClusterAuthorizationException' Cluster Authorization Failed
Fix Kafka ClusterAuthorizationException: diagnose missing CLUSTER ACLs, idempotent producer IdempotentWrite, transactional IDs, and admin operations on the cluster.
-
'Controller heartbeat timeout' Broker Fenced in KRaft
Fix Kafka KRaft 'controller heartbeat timeout / broker fenced': tune broker.heartbeat.interval.ms and broker.session.timeout.ms, and diagnose missed broker heartbeats.
-
'CoordinatorLoadInProgressException: The coordinator is loading' — Wait Out or Speed Up Offsets Loading
Fix CoordinatorLoadInProgressException in Kafka: why a broker is still loading __consumer_offsets after a restart and how to shorten the loading window.
-
'CoordinatorNotAvailableException' Group Coordinator Down
Fix Kafka CoordinatorNotAvailableException: resolve __consumer_offsets unavailability, coordinator load-in-progress, offline partitions, and under-replicated offsets topic.
-
'Corrupted index found' Offset Index Corrupted on Startup
Fix Kafka 'Corrupted index found' and 'Found invalid offset index' errors: understand index rebuilds on restart, time index corruption, and slow recovery startups.
-
'CorruptRecordException' Message Failed Its CRC Checksum
Fix Kafka CorruptRecordException: diagnose CRC32C checksum mismatches from network corruption, bad disks, truncated segments after unclean shutdown, and consumer fetch detection.
-
'Error for partition topic-0 at offset 12345' ReplicaFetcherThread Failure
Decode ReplicaFetcherThread errors when a Kafka follower can't fetch from the leader: NOT_LEADER, OFFSET_OUT_OF_RANGE, fetch size, and TLS causes.
-
'Error while fetching metadata' LEADER_NOT_AVAILABLE
Fix Kafka's 'Error while fetching metadata ... LEADER_NOT_AVAILABLE' and UNKNOWN_TOPIC_OR_PARTITION client warnings: causes, diagnostics, and resolution.
-
'Failed to allocate memory within the configured max blocking time' — Fixing Producer Buffer Exhaustion
Fix Kafka's producer 'Failed to allocate memory within the configured max blocking time' TimeoutException: diagnose buffer.memory exhaustion, slow brokers, and backpressure, then tune the producer to drain cleanly.
-
'Failed to append metadata record' Raft Append Failure
Fix KRaft 'Failed to append metadata record' to __cluster_metadata: diagnose lost leadership, no quorum, disk-full, and timeout failures on the Raft write path.
-
'FencedInstanceIdException' Static Member Has Been Fenced
Fix Kafka FencedInstanceIdException: why duplicate group.instance.id values fence a static consumer member, and how to keep static membership ids unique.
-
'Fetch request timed out' Consumer & Replica Fetch Timeout
Fix Kafka 'Fetch request timed out' / request.timeout.ms exceeded on fetch: resolve slow brokers, overlarge fetch sizes, network latency, and replica fetcher stalls.
-
'GroupAuthorizationException' Not Authorized to Access Group
Fix Kafka GroupAuthorizationException: diagnose missing group Read ACLs, wrong group.id, consumer principal mapping, and prefixed vs literal group patterns.
-
'GroupMaxSizeReachedException: Consumer group has reached maximum size' — Fix Oversized Consumer Groups
Fix GroupMaxSizeReachedException in Kafka: understand group.max.size limits, leaked or duplicate consumers, and right-size the consumer counts.
-
'IllegalGenerationException' Generation Is Not the Current Generation
Fix Kafka IllegalGenerationException: why a stale group generation rejects commits and heartbeats after a rebalance, and how to rejoin with the current generation.
-
'InvalidRecordException: One or more records have been rejected' Records Rejected by Broker Validation
Fix Kafka InvalidRecordException when the broker rejects records: null keys on compacted topics, out-of-range timestamps, bad magic bytes, and transactional misuse.
-
'java.net.SocketTimeoutException: Connection timed out' TCP Connect
Fix Kafka 'Connection timed out' at TCP connect — diagnose firewall DROP rules, security groups, and routing black holes, distinct from request.timeout.ms.
-
'LeaderNotAvailableException' Leader Election in Progress
Fix Kafka LeaderNotAvailableException: understand transient leader election on new topics, stale metadata, offline partitions, and when to retry vs investigate.
-
'left group due to expired session timeout' Consumer Drop
Fix Kafka consumers leaving the group on expired session timeout: tune session.timeout.ms and max.poll.interval.ms, cut GC pauses, and fix network and heartbeat stalls.
-
'NotCoordinatorException: This is not the correct coordinator' — Refresh the Group Coordinator
Fix NotCoordinatorException in Kafka: why a broker rejects a group request as the wrong coordinator, and how to fix stale metadata and offsets-topic issues.
-
'NotEnoughReplicasException: Messages are rejected' Fewer In-Sync Replicas Than Required
Resolve Kafka NotEnoughReplicasException and NotEnoughReplicasAfterAppendException: ISR dropped below min.insync.replicas under acks=all. Diagnose ISR and fix durability.
-
'NotLeaderOrFollowerException' Stale Leader Metadata
Fix Kafka NotLeaderOrFollowerException (formerly NotLeaderForPartition): stale client metadata after a leader move, reassignments, and broker restarts.
-
'OutOfOrderSequenceException' Out of Order Sequence Number
Fix Kafka OutOfOrderSequenceException: diagnose idempotent producer sequence gaps from dropped batches, message loss via unclean leader election, and PID resets; why it is non-recoverable and how to prevent it.
-
'PKIX path building failed' Unable to Find Valid Certification Path
Fix Kafka PKIX path building failed: diagnose missing CA in the truststore, incomplete chains, wrong truststore, bad_certificate alerts, and self-signed broker certs.
-
'ProducerFencedException' — Fixing Fenced Transactional Producers
Fix Kafka's ProducerFencedException when a transactional producer is fenced by a newer instance: diagnose duplicate transactional.id, zombie producers, timeouts, and restarts, then restore exactly-once safely.
-
'RecordDeserializationException: Error deserializing key/value for partition' — Handle the Poison Record
Fix RecordDeserializationException in Kafka: recover from poison records, deserializer and schema mismatches, and stuck partitions using DLQs.
-
'RecordTooLargeException' Message Exceeds max.request.size
Fix Kafka 'RecordTooLargeException: The message is N bytes ... larger than max.request.size': align max.request.size, message.max.bytes, max.message.bytes, fetch limits and compression.
-
'SaslAuthenticationException: Authentication failed' Invalid Credentials
Fix Kafka SaslAuthenticationException: diagnose bad SCRAM/PLAIN passwords, wrong JAAS config, missing mechanism, Kerberos keytab issues, and broker SASL setup.
-
'SerializationException: Error serializing Avro message' Error Serializing Message
Fix Kafka 'SerializationException: Error serializing Avro message': wrong key/value.serializer, type mismatches, and Schema Registry subject-not-found or incompatible-schema failures.
-
'Session expired for /controller' ZooKeeper Session Expiry
Fix Kafka ZooKeeper SessionExpiredException for /controller: diagnose long GC pauses, low session timeouts, lost ephemeral nodes, controller re-election, and clock skew.
-
'Snapshot generation failed' Metadata Snapshot Write Error
Fix KRaft 'Snapshot generation failed': diagnose disk-full, permissions, and I/O errors when the controller writes a __cluster_metadata snapshot to checkpoint state.
-
'SslAuthenticationException: SSL channel closed' Failed to Send SSL Close Message
Fix Kafka SSL channel closed: diagnose plaintext clients hitting SSL listeners, wrong security.protocol, abrupt connection drops, and proxy/LB TLS termination.
-
'SSLHandshakeException: Received fatal alert: handshake_failure'
Fix Kafka SSLHandshakeException handshake_failure: diagnose TLS version mismatch, cipher suite gaps, one-way vs mTLS, missing client cert, and protocol disablement.
-
'ThrottlingQuotaExceededException: The throttling quota has been exceeded' — Manage Client Quotas and Backoff
Fix ThrottlingQuotaExceededException in Kafka: understand controller-mutation and client quotas, why requests get throttled, and how to back off.
-
'TopicAuthorizationException' Not Authorized to Access Topics
Fix Kafka TopicAuthorizationException: diagnose missing topic Read/Write/Describe ACLs, principal mismatch, prefixed patterns, and metadata describe denials.
-
'Unable to fetch metadata log' Follower Far Behind
Fix KRaft 'Unable to fetch metadata log' / 'Unable to catch up to metadata log': diagnose a follower controller or broker lagging the __cluster_metadata leader.
-
'UnknownMemberIdException' Coordinator Is Not Aware of This Member
Fix Kafka UnknownMemberIdException: why the group coordinator evicts a consumer member id after session timeouts, and how to keep heartbeats alive to rejoin cleanly.
-
'UnknownProducerIdException' Producer ID Not Found by Broker
Fix Kafka UnknownProducerIdException: diagnose idempotent producer ID state evicted by short retention.ms, producer.id.expiration.ms expiry, and old-broker restarts; tune retention and adopt KIP-360.
-
'WakeupException' Thrown During Consumer Poll
Understand Kafka WakeupException: why consumer.wakeup() interrupts poll(), how to handle it for clean shutdown, and how to tell intended wakeups from real failures.
Kafka AI prompts
Copy-paste, production-safe prompts for this stack.
Kafka command center
Top errors, validators, and runbooks for the whole stack.
Frequently asked questions
Why do I get CommitFailedException in my Kafka consumer?
max.poll.records or the per-record work, or raise max.poll.interval.ms. See CommitFailedException.What does OffsetOutOfRangeException mean?
auto.offset.reset (earliest/latest/none). See OffsetOutOfRangeException.My producer keeps hitting TimeoutException — where do I start?
delivery.timeout.ms, request.timeout.ms, and acks. See producer TimeoutException.