🚨 Kafka Error Guide: 'RecordDeserializationException: Error deserializing key/value for partition' — Handle the Poison Record

Overview

RecordDeserializationException is thrown inside consumer.poll() when the configured deserializer cannot turn a record’s bytes into an object. The consumer cannot advance past the offending record, so the partition stalls:

org.apache.kafka.common.errors.RecordDeserializationException: Error deserializing key/value for partition orders-4 at offset 918273. If needed, please seek past the record to continue consumption.
Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id 42
Caused by: org.apache.avro.AvroTypeException: Found string, expecting long

Because poll() fails on that record, a single bad message — a “poison record” — blocks the entire partition until it is handled or skipped.

Symptoms

• A consumer throws RecordDeserializationException from poll() and stops making progress on one partition.
• Lag grows steadily on a single partition while others consume normally.
• Restarting the consumer immediately hits the same offset and fails again — a crash loop.
• The stack trace names Avro/Protobuf/JSON deserialization or a Schema Registry id.
• The error started right after a producer schema change or a wrong topic/serializer pairing.

Common Root Causes

• Schema/format mismatch — the producer wrote a format or schema the consumer’s deserializer does not expect (e.g., Avro bytes read by a JSON deserializer).
• Poison record — a single malformed or corrupt message among valid ones.
• Wrong deserializer configuration — key.deserializer/value.deserializer do not match how records were serialized.
• Schema Registry mismatch — the consumer cannot fetch or is incompatible with the schema id embedded in the record.
• Producer bug — a producer wrote plain bytes or a different type to a topic consumers expect to be schema-encoded.
• Wrong topic wiring — the consumer subscribed to a topic carrying a different payload type than configured.

Diagnostic Workflow

Identify the exact partition and offset from the exception, then inspect that record’s raw bytes:

kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic orders --partition 4 --offset 918273 --max-messages 1 \
  --property print.key=true --property print.timestamp=true

Check what serializer the producer actually used vs. the consumer’s deserializer config:

grep -RiE 'value.serializer|value.deserializer|key.serializer|key.deserializer' src/ config/

If using Schema Registry, look up the schema id embedded in the record:

curl -s http://schema-registry:8081/schemas/ids/42 | python3 -m json.tool

Confirm the current consumer position so you know where it is stuck:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group orders | grep 'orders *4'

Example Root Cause Analysis

An orders consumer group started crash-looping on RecordDeserializationException for partition 4 at a specific offset, with the cause AvroTypeException: Found string, expecting long. Lag on partition 4 climbed while the other partitions stayed healthy.

Reading the single record at that offset with kafka-console-consumer.sh and cross-checking the Schema Registry showed a producer had been deployed with a bug that wrote the order_id field as a string on one code path, violating the registered Avro schema that declared it a long. Only records from that buggy path were poison; the rest of the partition was fine. Every consumer restart replayed the same offset and died again.

The immediate recovery was to move the consumer past the poison record. Because the pipeline could tolerate skipping the malformed message, the team seeked the group past that offset (kafka-consumer-groups.sh --reset-offsets --shift-by) to unstick the partition, then rolled back the buggy producer. The durable fix was to add a DeserializationExceptionHandler (or an errors-tolerant deserializer routing to a dead letter topic) so a future poison record is dead-lettered with its partition/offset context instead of halting consumption. The root cause was a producer violating the schema contract, not a broker or consumer fault.

Prevention Best Practices

• Wrap consumers with a deserialization error handler (Streams DeserializationExceptionHandler, or an errors-tolerant deserializer) that routes poison records to a dead letter topic instead of stalling.
• Enforce Schema Registry compatibility (BACKWARD/FULL) in producer CI so a schema-violating producer never reaches production.
• Match key/value deserializers exactly to the producer’s serializers, and keep one payload type per topic.
• Capture partition, offset, and the raw bytes when dead-lettering so poison records are debuggable and replayable after a fix.
• Alert on single-partition lag growth — the classic signature of a stuck poison record.
• Validate producer output against the schema in tests so type mismatches are caught before publish.

Quick Command Reference

# Read the exact poison record
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders --partition 4 --offset 918273 --max-messages 1

# Look up an embedded schema id
curl -s http://schema-registry:8081/schemas/ids/42

# Current stuck position
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group orders

# Skip past a poison record (only if skipping is acceptable)
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group orders --topic orders:4 --reset-offsets --shift-by 1 --execute

Conclusion

RecordDeserializationException means a single record’s bytes could not be deserialized, and because it happens inside poll(), one poison record blocks the whole partition. The cause is a contract break: a schema/format mismatch, a wrong deserializer, or a producer bug. Recover by identifying the exact offset, skipping it if loss is acceptable, and fixing the producer. Prevent recurrence with an error handler that dead-letters poison records and Schema Registry compatibility enforcement so bad payloads never reach the topic.