🚨 Kafka Error Guide: 'Failed to allocate memory within the configured max blocking time' — Fixing Producer Buffer Exhaustion

Overview

A Kafka producer starts failing send() calls with a TimeoutException complaining that it could not allocate buffer memory. The producer’s record accumulator (sized by buffer.memory) is full, and a new record waited longer than max.block.ms for space to free up:

org.apache.kafka.common.errors.TimeoutException: Failed to allocate memory within the configured max blocking time 60000 ms.

You may also see the closely related variant when the producer blocks waiting for topic metadata while the buffer is under pressure:

org.apache.kafka.common.errors.TimeoutException: Topic myeventstopic not present in metadata after 60000 ms.

This is backpressure surfacing as an error: the application is trying to produce faster than the producer can hand records off to the brokers, and the in-memory buffer has filled to its limit.

Symptoms

• producer.send(...) (or the returned future) fails with Failed to allocate memory within the configured max blocking time.
• Throughput collapses in bursts — the producer runs fine, then stalls, then throws — tracking traffic spikes.
• The producer JVM shows the sender thread busy while record-queue-time-avg and buffer-available-bytes metrics show the accumulator saturated.
• Errors coincide with broker-side slowness: rising produce latency, under-replicated partitions, or acks=all waiting on a lagging ISR.
• Large batches or large individual records drain the buffer faster than the sender can flush.

Common Root Causes

• Undersized buffer.memory for the produce rate. The default 32 MB accumulator cannot hold the in-flight backlog when the app bursts faster than the sender drains.
• Slow or overloaded brokers. High produce latency (busy disks, acks=all with a slow follower, under-replicated partitions) means batches leave the buffer slowly, so it stays full.
• max.block.ms too short for the load pattern. Legitimate short bursts exceed the blocking window and throw instead of waiting out the spike.
• Network or leader problems. Packet loss, throttling, or frequent leader changes stall the sender so the buffer backs up.
• Oversized records or batches. Large messages (or a big batch.size/linger.ms combination) consume buffer space faster than it is reclaimed.
• Too few max.in.flight.requests.per.connection combined with high latency. The sender cannot keep enough requests outstanding to drain the buffer at the produce rate.
• A single stuck partition. One offline or lagging partition’s records pile up in the accumulator and starve the whole producer.

Diagnostic Workflow

1. Confirm it is buffer exhaustion, not metadata. The “Failed to allocate memory” text is specifically the accumulator being full; the “not present in metadata” variant is a metadata/connectivity problem. Treat them separately.

2. Check broker-side produce health. If brokers are slow, the buffer will always refill. Look for under-replicated partitions and offline partitions:

kafka-topics.sh --bootstrap-server broker:9092 --describe --under-replicated-partitions
kafka-topics.sh --bootstrap-server broker:9092 --describe --unavailable-partitions

3. Measure end-to-end produce latency with the built-in perf tool to see whether the brokers, not the app, are the bottleneck:

kafka-producer-perf-test.sh \
  --topic myeventstopic \
  --num-records 100000 \
  --record-size 1024 \
  --throughput -1 \
  --producer-props bootstrap.servers=broker:9092 acks=all

4. Inspect the target topic for offline leaders or shrunken ISR that would stall the sender:

kafka-topics.sh --bootstrap-server broker:9092 --describe --topic myeventstopic

5. Read the producer’s own metrics. In the app, watch buffer-available-bytes, bufferpool-wait-ratio, record-queue-time-avg, and request-latency-avg — a wait-ratio near 1.0 with near-zero available bytes confirms the accumulator is the constraint.

6. Check broker request handler saturation. If broker request-handler idle percent is low, the cluster cannot absorb the produce rate:

kafka-run-class.sh kafka.tools.JmxTool \
  --object-name kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent \
  --jmx-url service:jmx:rmi:///jndi/rmi://broker:9999/jmxrmi --one-time true

Example Root Cause Analysis

An ingestion service producing clickstream events ran fine at steady state but threw Failed to allocate memory within the configured max blocking time 60000 ms every day during the evening peak. Producer metrics showed buffer-available-bytes hitting zero and bufferpool-wait-ratio pinned near 1.0 during the failures, while request-latency-avg on the producer tripled at the same moments.

Broker inspection explained the latency: --describe --under-replicated-partitions listed several partitions on the event topic, and one follower broker had a saturated disk. With acks=all, every batch waited for the slow follower to catch up, so batches left the accumulator slowly. The 32 MB default buffer.memory filled within seconds of the evening burst, and new send() calls blocked past max.block.ms and threw.

The immediate mitigation was to raise buffer.memory to 128 MB and increase max.block.ms so short bursts could ride out broker latency. The real fix was broker-side: replacing the degraded disk cleared the under-replicated partitions and dropped produce latency, after which the buffer drained normally even at peak. The team also added producer buffer and broker ISR alerts so the next disk degradation is caught before it becomes producer errors.

Prevention Best Practices

• Size buffer.memory to the burst, not the average. Give the accumulator enough headroom to absorb peak produce rate times worst-case broker latency.
• Keep brokers healthy. Alert on under-replicated and offline partitions and on produce latency; a slow broker becomes producer buffer exhaustion downstream.
• Set max.block.ms deliberately. Long enough to ride out normal bursts, but bounded so a genuinely broken cluster surfaces fast instead of blocking the app forever.
• Tune batching and compression. Reasonable batch.size, linger.ms, and compression.type improve throughput per byte of buffer.
• Match acks to durability need. acks=all is safest but couples the producer to the slowest ISR follower; ensure min.insync.replicas and replica health support it.
• Apply backpressure in the app. When send() futures pile up, throttle upstream rather than pushing an unbounded backlog into a fixed-size buffer.
• Watch producer buffer metrics. Alert on bufferpool-wait-ratio and buffer-available-bytes so saturation is visible before it throws.

Quick Command Reference

# Under-replicated and unavailable partitions (broker-side stalls)
kafka-topics.sh --bootstrap-server broker:9092 --describe --under-replicated-partitions
kafka-topics.sh --bootstrap-server broker:9092 --describe --unavailable-partitions

# Describe the target topic's leaders and ISR
kafka-topics.sh --bootstrap-server broker:9092 --describe --topic myeventstopic

# Benchmark real produce latency/throughput
kafka-producer-perf-test.sh --topic myeventstopic --num-records 100000 \
  --record-size 1024 --throughput -1 \
  --producer-props bootstrap.servers=broker:9092 acks=all

# Check broker request-handler idle percent via JMX
kafka-run-class.sh kafka.tools.JmxTool \
  --object-name kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent \
  --jmx-url service:jmx:rmi:///jndi/rmi://broker:9999/jmxrmi --one-time true

# Inspect effective producer/topic configs
kafka-configs.sh --bootstrap-server broker:9092 --describe --entity-type topics --entity-name myeventstopic

Conclusion

Failed to allocate memory within the configured max blocking time is Kafka producer backpressure made visible: records arrive faster than the sender can flush them, the fixed-size buffer.memory accumulator fills, and a new record waits past max.block.ms before failing. The buffer setting is only half the story — the buffer stays full when brokers are slow, so always check under-replicated partitions, produce latency, and ISR health alongside producer metrics. Right-size buffer.memory and max.block.ms for your burst pattern to buy headroom, but fix the underlying broker or network bottleneck so the accumulator drains cleanly at peak load.