🚨 Kafka Error Guide: 'RecordTooLargeException' Message Exceeds max.request.size

Exact Error Message

The client-side variant is thrown synchronously from send() (or delivered to the callback) before the record ever leaves the JVM:

org.apache.kafka.common.errors.RecordTooLargeException: The message is 2097243 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration.
	at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:...)
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:...)
	at com.acme.OrderProducer.publish(OrderProducer.java:...)

There is also a broker-side variant. Here the client passed its own check, but a broker rejected the produce request and the failure arrives in the callback:

[2026-06-29 14:02:11,883] ERROR Error when sending message to topic orders (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept.

The two are distinct: the first references max.request.size (a producer config) and the byte counts 2097243 > 1048576; the second is the broker enforcing message.max.bytes (or a topic’s max.message.bytes) and never mentions a client config value.

What the Error Means

Kafka enforces message-size limits at multiple layers, and RecordTooLargeException is the umbrella error for “this record/batch is too big for the limit at this layer.”

• Producer layer (max.request.size, default 1048576 = 1 MiB). Checked in KafkaProducer.ensureValidRecordSize() against the serialized size of a single record (key + value + headers + overhead). This is a hard pre-flight check; the record is never sent, and the byte counts in the message come straight from this comparison.
• Producer batch layer (batch.size / buffer.memory). A single record larger than batch.size still gets its own batch, so batch.size does not cause this error — but it interacts with throughput.
• Broker layer (message.max.bytes, default ~1 MiB). The broker validates the batch (a compressed record batch) it receives. If it exceeds message.max.bytes, it returns MESSAGE_TOO_LARGE.
• Topic layer (max.message.bytes). A per-topic override of the broker limit. A topic can be more or less permissive than the broker default; the topic value wins for that topic.
• Consumer layer (fetch.max.bytes, max.partition.fetch.bytes). If you raise the produce/broker limits but leave consumer fetch limits at defaults, consumers may be unable to read the large records (modern clients return at least one record even if it exceeds the fetch limit, but throughput and memory still suffer).

Because the producer measures the serialized size, compression matters: max.request.size is checked against uncompressed serialized bytes per record, whereas the broker’s message.max.bytes is checked against the compressed batch. A payload can pass the broker check after compression yet still fail the client check.

Common Causes

1. A genuinely oversized payload — embedding a file, image, or large JSON blob directly in the message value.
2. max.request.size left at the 1 MiB default while the application routinely produces multi-megabyte records.
3. Mismatched layers — you raised max.request.size on the producer but never raised message.max.bytes on the broker or max.message.bytes on the topic, so the broker rejects what the client accepted.
4. Headers and key overhead pushing a “1 MB” value just over the limit once serialized with metadata.
5. Schema/serialization bloat — verbose serialization (e.g. JSON instead of Avro/Protobuf) inflating the serialized size.
6. Consumer side under-provisioned — large records produced fine but max.partition.fetch.bytes too small, causing downstream stalls (a related symptom, not the same exception).

How to Reproduce the Error

With defaults, produce a value just over 1 MiB:

Properties props = new Properties();
props.put("bootstrap.servers", "kafka-1.internal:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.ByteArraySerializer");
// max.request.size defaults to 1048576

try (KafkaProducer<String, byte[]> p = new KafkaProducer<>(props)) {
    byte[] big = new byte[2 * 1024 * 1024]; // ~2 MiB
    p.send(new ProducerRecord<>("orders", "k1", big)).get(); // throws RecordTooLargeException
}

To reproduce the broker-side variant, raise the producer limit but leave the broker/topic at default:

max.request.size=5242880
compression.type=none

The client now accepts the 2 MiB record but the broker rejects it with “larger than the max message size the server will accept.”

Diagnostic Commands

Check the topic-level size override (if any) and other topic configs:

kafka-configs.sh --bootstrap-server kafka-1.internal:9092 --describe --entity-type topics --entity-name orders

Check the broker-wide message.max.bytes and replica fetch limit:

kafka-configs.sh --bootstrap-server kafka-1.internal:9092 --describe --entity-type brokers --entity-name 1

Confirm the topic and its replication/ISR are healthy:

kafka-topics.sh --bootstrap-server kafka-1.internal:9092 --describe --topic orders

Find the exact byte counts and which layer rejected the record in your logs:

grep -E "RecordTooLargeException|max.request.size|max message size" /var/log/orders-producer/app.log

Check the broker log for MESSAGE_TOO_LARGE rejections:

journalctl -u kafka --since "2026-06-29 14:00:00" | grep -iE "too large|message.max.bytes"

Step-by-Step Resolution

1. Decide whether large messages are appropriate. For payloads in the multi-MB range, prefer the claim-check pattern (store the blob in object storage, send a reference). Raising limits cluster-wide has real memory and latency cost.

2. If you must allow larger records, raise every layer consistently. The limits must satisfy max.request.size <= message.max.bytes (or topic max.message.bytes) <= replica.fetch.max.bytes, and consumers must allow the fetch:

   # producer
   max.request.size=5242880
   # broker (server.properties) — or per-topic max.message.bytes
   message.max.bytes=5242880
   replica.fetch.max.bytes=5242880
   # consumer
   fetch.max.bytes=52428800
   max.partition.fetch.bytes=5242880

3. Enable compression where it helps. compression.type=lz4 (or zstd) shrinks the batch the broker sees. This eases the broker/topic limit but does not relax the client max.request.size check, which is per-record on uncompressed serialized bytes.

4. Trim serialization overhead. Switch verbose JSON to Avro/Protobuf, drop redundant headers, and avoid embedding base64 blobs.

5. Verify consumers can read it. After raising produce limits, confirm max.partition.fetch.bytes and fetch.max.bytes are large enough that consumers do not stall or OOM.

6. Roll changes carefully. Broker and replica fetch settings affect the whole cluster’s memory footprint; change them deliberately and monitor heap and request latency.

Prevention and Best Practices

• Set message-size limits as a deliberate, documented policy across producer, broker, topic, and consumer — not ad hoc on one side.
• Default to the claim-check pattern for anything over a few hundred KB; Kafka is a log, not a blob store.
• Pick a compact serialization format and measure the serialized size, since that is what max.request.size checks.
• Add a pre-send size guard in application code so oversized records are dead-lettered with context instead of crashing a batch.
• When a size mismatch causes a production incident, paste the byte counts and your describe output into the incident assistant, and consult the broader Kafka guides for replication and consumer-fetch tuning.

Related Errors

• TimeoutException — if oversized batches are slow to replicate, you can also see delivery timeouts; rule out size limits first since RecordTooLargeException fails fast.
• InvalidRecordException — another broker-side validation failure (e.g. bad timestamps or compaction key rules) that, like the broker variant here, rejects the produce request rather than the client.
• SerializationException — happens earlier in the pipeline; a record must serialize successfully before its serialized size can be checked against max.request.size.

Frequently Asked Questions

Does enabling compression fix RecordTooLargeException? It depends which variant. Compression shrinks the batch the broker validates against message.max.bytes, so it can fix the broker-side rejection. It does not fix the client-side max.request.size error, which is measured per record on uncompressed serialized bytes before batching.

Why does the producer say 2097243 bytes but my value is only 2 MB? The serialized size includes the key, value, record headers, and record/batch overhead. A “2 MB” value can serialize to slightly more, and a value just under a limit can tip over once metadata is added.

I raised max.request.size but still get rejected — why? You only fixed the client layer. The broker still enforces message.max.bytes and the topic may have a stricter max.message.bytes. Raise all of them, plus replica.fetch.max.bytes, or the broker (or followers) will refuse the record.

Do I need to change consumer settings too? Usually yes. If you produce records larger than max.partition.fetch.bytes, consumers can stall or behave inefficiently. Raise fetch.max.bytes and max.partition.fetch.bytes to match the new maximum record size.

Is raising the limits to allow 50 MB messages a good idea? Generally no. Large messages increase broker heap pressure, replication latency, and consumer memory use. For large payloads, use the claim-check pattern and keep Kafka records small and uniform.