🚨 Kafka Error Guide: 'CorruptRecordException' Message Failed Its CRC Checksum

Exact Error Message

org.apache.kafka.common.errors.CorruptRecordException: This message has failed its CRC checksum, exceeds the valid size, has a null key for compacted topic, or is otherwise corrupt.
	at org.apache.kafka.common.record.DefaultRecordBatch.ensureValid(DefaultRecordBatch.java:168)
	at org.apache.kafka.clients.consumer.internals.CompletedFetch.maybeEnsureValid(CompletedFetch.java:283)
	at org.apache.kafka.clients.consumer.internals.CompletedFetch.fetchRecords(CompletedFetch.java:326)
	at org.apache.kafka.clients.consumer.internals.AbstractFetch.fetchRecords(AbstractFetch.java:344)
	at org.apache.kafka.clients.consumer.internals.AbstractFetch.collectFetch(AbstractFetch.java:306)
	at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1267)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1226)

A broker-side variant frequently appears in server.log when the log subsystem validates a segment on read or recovery:

[2026-06-29 04:11:52,007] ERROR [ReplicaFetcher replicaId=3, leaderId=1, fetcherId=0] Found invalid messages during fetch for partition orders-7 offset 84512337 (kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.CorruptRecordException: Record size 1684827238 exceeds the largest allowable message size (1048588).

What the Error Means

Every Kafka record batch carries a CRC32C checksum computed over the batch header and payload. The checksum is calculated once by the producer when the batch is serialized and is never recomputed in flight, which lets brokers and consumers cheaply detect bit-level corruption end to end. CorruptRecordException is thrown whenever a component reads a batch whose stored CRC does not match the CRC recomputed over the bytes it actually received, or whose framing is structurally impossible: a declared batch length larger than the remaining bytes, a record count that overruns the buffer, or a null key on a record destined for a compacted (cleanup.policy=compact) topic.

The critical point is that this is a data integrity error, not a transient network or load condition. The bytes on disk or in the fetch response are wrong. Kafka deliberately refuses to hand corrupt data to your application rather than silently returning garbage. Because the checksum is stamped at produce time, the corruption could have been introduced anywhere along the path: in the producer’s memory before the CRC was computed (rare), on the wire during transmission, on the broker’s disk, or during a fetch from the leader. Where it surfaces tells you where to look.

Common Causes

• Network corruption on the wire: a failing NIC, a faulty switch port, bad optics, or a marginal cable can flip bits in flight. TCP checksums catch most single-bit errors but are only 16 bits wide and miss a meaningful fraction of multi-bit corruption, especially at high throughput. Non-ECC memory on intermediate hops compounds this.
• Disk or filesystem corruption on the broker log dir: bad sectors, a controller with a volatile write cache and no battery backup, silent bit rot, or filesystem metadata damage can alter committed segment bytes after they were written cleanly.
• Truncated or partially written segments after an unclean shutdown: a power loss or kill -9 mid-write can leave a segment with a half-written final batch. On restart, log recovery may flag the trailing batch as corrupt.
• Memory corruption (non-ECC RAM): a bit flip in the page cache or in the producer’s accumulator before the CRC is stamped produces data that is internally consistent at one hop but fails downstream.
• Consumer-side detection during fetch: the most common place users see it. The consumer recomputes the CRC on the fetched batch and rejects it, even though the root cause is upstream on the broker or network.

How to Reproduce the Error

Genuine corruption is hard to force safely, but you can demonstrate the detection path. A minimal idempotent producer writing to a compacted topic with a null key triggers the “null key for compacted topic” branch of the same exception class:

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

try (Producer<String, String> producer = new KafkaProducer<>(props)) {
    // Target topic has cleanup.policy=compact; a null key is invalid here.
    producer.send(new ProducerRecord<>("compacted-config", null, "payload"))
            .get(); // broker rejects with a corrupt/invalid record error
}

To observe true CRC failures in a lab, you can corrupt a closed .log segment file on a single broker with a hex editor while the broker is stopped, flip a byte in the payload region, and restart. The broker’s recovery or a consumer fetch will then surface CorruptRecordException for the offending offset. Never do this outside a disposable test cluster.

Diagnostic Commands

Inspect the topic and confirm whether it is compacted (which changes the null-key rule):

kafka-topics.sh --bootstrap-server broker:9092 --describe --topic orders
kafka-configs.sh --bootstrap-server broker:9092 --describe \
  --entity-type topics --entity-name orders

Identify which consumer group and partition is stuck on the bad offset:

kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --describe --group order-processors

Search broker logs for the corruption signature and the affected partition/offset:

grep -i "CorruptRecord\|invalid messages\|Found invalid" \
  /var/log/kafka/server.log

journalctl -u kafka --since "1 hour ago" | grep -i corrupt

Check the host for the underlying hardware story behind disk or memory corruption:

dmesg | grep -iE "i/o error|ata|nvme|ecc|mce|bad sector"
cat /sys/block/sda/device/model 2>/dev/null

Step-by-Step Resolution

1. Locate the exact partition and offset. The exception or grep output names them. This tells you which broker holds the leader replica and which segment file is suspect.
2. Determine the blast radius. If the same offset is corrupt on every replica, the corruption likely predates replication (producer memory or network at produce time, or it was replicated from a bad leader). If only one broker is affected, it is local disk or memory on that host.
3. Heal from a clean replica. If a healthy replica exists, the cleanest fix is to remove the corrupt broker’s copy of that partition and let it re-replicate from the leader. Stop the affected broker, delete its replica directory for that partition, and restart so it refetches a clean copy. The broker is the authority here; you are not mutating data via the CLI.
4. If all replicas are corrupt, the affected offset(s) are unrecoverable. A consumer can skip the poisoned offset by advancing its committed offset past the bad record so processing resumes; the corrupt record itself is lost.
5. Fix the hardware root cause. Replace the failing NIC, switch port, cable, or disk indicated by dmesg/SMART. Insist on ECC RAM on brokers. A broker that produces local corruption will keep producing it.
6. Right-size message limits. When the error reads “exceeds the largest allowable message size,” it is framing corruption, not a genuinely oversized record; do not blindly raise message.max.bytes. Confirm the bytes are actually corrupt before treating it as a sizing problem. Target broker values stay conservative:

message.max.bytes=1048588
replica.fetch.max.bytes=1048576

Prevention and Best Practices

Run brokers on ECC memory and enterprise drives with power-loss-protected write caches; consumer-grade SSDs with volatile caches are a leading cause of post-shutdown segment corruption. Enable end-to-end checksums in your storage stack (ZFS or btrfs scrubbing) so silent bit rot is caught before Kafka reads the bad bytes. Keep replication.factor=3 with min.insync.replicas=2 so a single corrupt replica is never the only copy. Configure graceful shutdowns (controlled.shutdown.enable=true) and adequate shutdown timeouts so brokers flush and close segments cleanly instead of leaving truncated tails. Monitor dmesg, SMART, and NIC error counters; corruption is almost always preceded by rising hardware error rates. Finally, leave the producer’s idempotence and CRC validation on so corruption is caught at the earliest possible hop rather than propagating into downstream systems. For a live walkthrough of a corruption incident, the incident assistant can correlate broker logs with host hardware events.

Related Errors

CorruptRecordException is in the same broker validation family as InvalidRecordException, which is thrown for records that are structurally well-formed but semantically rejected (bad compression, invalid timestamps), versus a CRC or framing failure here. A RecordTooLargeException is distinct: it is a clean record that genuinely exceeds limits, whereas the “exceeds valid size” branch of CorruptRecordException is a corrupted length field. If corruption on the wire is intermittent, you may also see NetworkException from the producer when connections drop around the same faulty hardware. For more, see the full set of Kafka guides.

Frequently Asked Questions

Is CorruptRecordException safe to retry?

No. Unlike a TimeoutException, this is not transient. Retrying a fetch against the same corrupt offset on the same replica returns the same bad bytes. You must heal from a clean replica or skip the offset; retrying alone never resolves it.

Why does my consumer see the error when the producer reported success?

Because the CRC is stamped at produce time and only re-verified on read. The producer’s success means the batch was accepted; corruption introduced afterward on disk or during replication only surfaces when the consumer recomputes the checksum during fetch.

Can I disable CRC checking to get past the bad record?

Modern clients do not expose a switch to skip CRC validation, and you should not want one. The check is the only thing standing between your application and silently corrupted data. Skip the specific offset instead by advancing the committed offset past it.

Does a higher replication factor prevent corruption?

It does not prevent the underlying bit flips, but it dramatically improves recoverability. With three replicas, a single host’s disk or memory corruption affects one copy, and Kafka can re-replicate a clean version from the others. With one replica, the corrupt record is simply lost.

How do I tell network corruption from disk corruption?

Look at the blast radius and timing. Disk or memory corruption is sticky and tied to one broker’s segment files, and dmesg usually shows I/O or MCE events. Network corruption is intermittent, correlates with throughput spikes, often coincides with NetworkException on producers, and clears when the bad NIC, cable, or switch port is swapped.