🚨 Kafka Error Guide: 'OutOfMemoryError: Java heap space' — Size the Broker Heap

Overview

When the Kafka JVM cannot allocate an object because the heap is exhausted, it throws OutOfMemoryError: Java heap space. On a broker this typically kills a critical thread and leaves the broker in a broken state:

ERROR [ReplicaManager broker=2] Error processing fetch operation (kafka.server.ReplicaManager)
java.lang.OutOfMemoryError: Java heap space
        at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:64)
        at org.apache.kafka.common.record.MemoryRecords.readableRecords(MemoryRecords.java:59)

By default Kafka is often launched so that a heap OOM triggers a hard exit, so you may also see it in the process/journal logs right before the broker dies:

Terminating due to java.lang.OutOfMemoryError: Java heap space

Symptoms

• Broker or client logs java.lang.OutOfMemoryError: Java heap space and threads die or the JVM exits.
• The broker crashes and restarts (often in a loop) under load, leaving partitions under-replicated or briefly offline.
• Long GC pauses and rising heap usage precede the crash; the broker becomes unresponsive before dying.
• Producers/consumers time out or disconnect around the crash window.
• A heap dump file (java_pid<pid>.hprof) appears if HeapDumpOnOutOfMemoryError is enabled.

Common Root Causes

• Heap set too large, starving the page cache. Kafka relies on OS page cache for throughput; a giant heap steals that RAM and increases GC pressure without helping.
• Heap set too small for the workload. Too many partitions, connections, or in-flight requests exceed a tiny heap.
• Oversized fetch/replica buffers. Large replica.fetch.max.bytes, socket.request.max.bytes, or per-partition fetch sizes multiplied across many partitions blow up transient allocations.
• Huge message batches. Large message.max.bytes / max.message.bytes combined with many concurrent requests.
• Connection storms. Thousands of client connections each holding buffers.
• Client-side OOM. A consumer with large max.partition.fetch.bytes x many partitions, or a producer with a big buffer.memory, running in a small JVM.

Diagnostic Workflow

First confirm the crash is a heap OOM and capture the broker’s current heap settings:

grep -i 'OutOfMemoryError\|Java heap space' /var/lib/kafka/logs/server.log
journalctl -u kafka --since '30 min ago' | grep -i 'outofmemory\|terminating'

KPID=$(pgrep -f kafka.Kafka)
jcmd $KPID VM.flags | tr ' ' '\n' | grep -iE 'Xmx|Xms|MaxHeap|InitialHeap'

Watch live heap and GC behavior to see whether it is a leak (grows and never recovers) or genuine pressure:

jstat -gcutil $KPID 5s 12          # O and gc columns climbing toward 100%
jcmd $KPID GC.heap_info

Check the workload that drives allocations — partition count and the relevant broker limits:

kafka-topics.sh --bootstrap-server localhost:9092 --describe \
  --under-replicated-partitions

kafka-configs.sh --bootstrap-server localhost:9092 --describe \
  --entity-type brokers --entity-name 2 \
  | grep -Ei 'replica.fetch.max.bytes|socket.request.max.bytes|message.max.bytes'

Confirm how much RAM the host actually has so heap and page cache can be balanced:

free -h

Example Root Cause Analysis

A broker on a 32 GB host began crash-looping with OutOfMemoryError: Java heap space inside ReplicaManager fetch handling. jcmd $KPID VM.flags showed -Xmx28g — nearly the whole machine had been handed to the JVM heap, leaving almost nothing for the OS page cache Kafka depends on. jstat -gcutil showed the old generation pinned near 100% with back-to-back full GCs just before each crash.

Digging into config, replica.fetch.max.bytes and socket.request.max.bytes had both been raised well above defaults, and the broker hosted several thousand partitions. Each replica fetch allocated large transient buffers, and across that many partitions the concurrent fetch load produced allocation spikes the oversized-but-GC-thrashing heap couldn’t service.

The fix was two-fold: reduce -Xmx to 6 GB (a typical broker heap, leaving the bulk of RAM for page cache), and bring the fetch/socket byte limits back toward sane values. After the change, jstat showed old-gen settling comfortably below the ceiling, GC pauses shrank, page-cache-driven throughput recovered, and the broker stopped OOM-ing. The root cause was heap misconfiguration — too large, starving cache and thrashing GC — compounded by oversized fetch buffers, not a genuine memory leak.

Prevention Best Practices

• Set a moderate broker heap (commonly ~5-6 GB via KAFKA_HEAP_OPTS="-Xmx6g -Xms6g") and leave the rest of RAM for the OS page cache; bigger is not better.
• Set -Xms equal to -Xmx to avoid heap resizing pauses, and enable -XX:+HeapDumpOnOutOfMemoryError so you can diagnose the next event.
• Keep replica.fetch.max.bytes, socket.request.max.bytes, and message.max.bytes at sane values; remember they multiply across partitions and connections.
• Cap partitions and connections per broker during capacity planning so transient allocations stay bounded.
• Monitor JVM heap utilization and GC pause time; alert before the heap saturates rather than after the crash.
• On clients, size max.partition.fetch.bytes and producer buffer.memory against the client JVM’s heap and its topic/partition fan-out.

Quick Command Reference

# Confirm it's a heap OOM and read heap flags
grep -i 'Java heap space' /var/lib/kafka/logs/server.log
KPID=$(pgrep -f kafka.Kafka)
jcmd $KPID VM.flags | tr ' ' '\n' | grep -iE 'Xmx|Xms'

# Live heap / GC behavior
jstat -gcutil $KPID 5s 12
jcmd $KPID GC.heap_info

# Workload and buffer sizing
kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions
kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type brokers --entity-name 2

# Host RAM (balance heap vs. page cache)
free -h

Conclusion

OutOfMemoryError: Java heap space on a Kafka broker is almost always a sizing problem, not a leak: the heap is either too large (starving the page cache and thrashing GC) or too small for the partition, fetch, and connection load. Diagnose it by confirming the OOM, reading the actual -Xmx, and watching heap/GC with jstat, then right-size the heap to leave ample RAM for page cache and rein in oversized fetch and message buffers. Heap and GC alerting turns the next crash into a warning you can act on first.