🚨 Kafka Error Guide: 'CoordinatorLoadInProgressException: The coordinator is loading' — Wait Out or Speed Up Offsets Loading

Overview

CoordinatorLoadInProgressException is returned when a broker has just become the coordinator for a consumer group but is still reading that group’s state from __consumer_offsets and cannot yet serve requests:

org.apache.kafka.common.errors.CoordinatorLoadInProgressException: The coordinator is loading and hence can't process requests for this group.
[2026-07-09 09:31:07] INFO [GroupCoordinator 2] Loading group metadata for orders with generation 42 (kafka.coordinator.group.GroupCoordinator)

It is a retriable, expected condition right after a broker restart or coordinator move. Clients retry until loading finishes. It only signals a real problem when loading takes minutes because the offsets partition is huge.

Symptoms

• Consumers cannot join groups or commit offsets for tens of seconds after a broker restart.
• Broker logs show Loading group metadata / Finished loading offsets and group metadata bracketing the delay.
• Rebalances stall until the coordinator finishes loading, then complete normally.
• The error clears on its own — clients that retry recover without operator action.
• Loading windows grow over time as the __consumer_offsets partitions accumulate data.

Common Root Causes

• Recent broker restart or failover — the new coordinator must replay the group’s __consumer_offsets partition before serving it.
• Large __consumer_offsets partitions — many groups, high commit frequency, or poor compaction make the partition slow to load.
• Compaction lagging on the offsets topic — the log cleaner is behind, so the coordinator reads far more records than the live state requires.
• Slow disk — reading the offsets partition from a slow or contended volume stretches the loading window.
• Frequent coordinator moves — repeated leadership churn forces repeated reloads.

Diagnostic Workflow

Measure how long loading takes by reading the broker’s coordinator log lines:

grep -iE 'Loading group metadata|Finished loading' /var/log/kafka/server.log | tail -20

Check the on-disk size of the __consumer_offsets partitions (large = slow load):

kafka-log-dirs.sh --bootstrap-server localhost:9092 \
  --describe --topic-list __consumer_offsets | python3 -m json.tool | grep -A2 size

Confirm the log cleaner is running and not backlogged on the offsets topic:

grep -iE 'log-cleaner|cleaner' /var/log/kafka/log-cleaner.log | tail
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name __consumer_offsets --describe

Check whether the group has recovered after waiting:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group orders --state

Example Root Cause Analysis

After a routine broker restart, one team reported consumers stuck for over two minutes before they could commit, all logging CoordinatorLoadInProgressException. Other groups recovered in seconds, so the size difference was the clue.

kafka-log-dirs.sh showed the __consumer_offsets partition owning that group was tens of gigabytes — far larger than peers. The log-cleaner log showed the cleaner had been disabled months earlier during an unrelated incident and never re-enabled, so the offsets partition had never been compacted and held millions of superseded commit records. The coordinator had to replay all of them on every failover.

Re-enabling log.cleaner.enable=true and letting compaction run reduced the partition from tens of GB back to a few hundred MB. Subsequent restarts loaded the coordinator in under two seconds. Nothing was wrong with the clients — they had retried correctly the whole time — the fix was restoring compaction on the offsets topic.

Prevention Best Practices

• Keep log.cleaner.enable=true and ensure log.cleaner.threads is sufficient so __consumer_offsets stays compacted and small.
• Monitor __consumer_offsets partition sizes and alert on unexpected growth — the leading cause of long load times.
• Place Kafka log directories on fast disks so offsets replay is quick during failover.
• Roll brokers one at a time and expect a brief, self-healing burst of this error per restart.
• Treat this exception as retriable in application code and let the client library retry rather than surfacing it as a hard failure.

Quick Command Reference

# How long loading took
grep -iE 'Loading group metadata|Finished loading' /var/log/kafka/server.log | tail

# Offsets partition size (large = slow load)
kafka-log-dirs.sh --bootstrap-server localhost:9092 --describe --topic-list __consumer_offsets

# Offsets topic config (cleanup.policy should be compact)
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name __consumer_offsets --describe

# Group state after waiting
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group orders --state

Conclusion

CoordinatorLoadInProgressException is Kafka telling a client to wait while a newly assigned coordinator replays a group’s state from __consumer_offsets. As a brief post-restart event it is normal and self-healing. When loading drags on for minutes, the offsets partition has grown too large — usually because compaction stalled. Keep the log cleaner running, monitor offsets-partition size, and use fast disks so coordinator loading stays measured in seconds, not minutes.