🐰 RabbitMQ Error: Quorum/Stream Queue Leaders Not Balancing Across Nodes

Exact Error Message

There is no exception — the failure mode is a skewed cluster where every leader lands on one node. rabbitmq-queues makes the imbalance obvious:

$ rabbitmq-queues quorum_status orders
Leader: rabbit@node1
Members:
    rabbit@node1     (leader)
    rabbit@node2
    rabbit@node3

Across the whole cluster the pattern repeats — node1 leads everything:

$ rabbitmqctl list_queues name type leader --formatter pretty_table
┌───────────────┬────────┬──────────────┐
│ name          │ type   │ leader       │
├───────────────┼────────┼──────────────┤
│ orders        │ quorum │ rabbit@node1 │
│ payments      │ quorum │ rabbit@node1 │
│ events        │ stream │ rabbit@node1 │
│ audit         │ quorum │ rabbit@node1 │
└───────────────┴────────┴──────────────┘

node1 is saturated on CPU and disk I/O while node2 and node3 sit idle.

What It Means

Every quorum queue and stream has a single leader replica that handles all publishes and consumer coordination; the other replicas are followers. Where the leader is placed is decided at queue-declaration time by the queue leader locator strategy. If that strategy is client-local (the historical default in older versions) and all your queues are declared through connections to the same node, every leader is created on that one node.

The result is not an error but a hot node: one broker does all the work while its peers idle. The fix is to choose a balanced locator so new leaders spread across nodes, then rebalance the leaders that already exist.

Common Causes

• queue_leader_locator is set to client-local and all declarations arrive on one node (common behind a load balancer that pins connections).
• Queues were declared before the locator was changed, so their leaders never moved.
• A node was restarted; its queues failed over to a surviving node and leadership was never rebalanced afterward.
• Applications reconnect to a single “primary” endpoint instead of spreading across the cluster.
• A per-queue x-queue-leader-locator argument overrides the cluster-wide default with client-local.

Diagnostic Commands

Check the cluster-wide default locator:

rabbitmqctl environment | grep -A2 queue_leader_locator

Or read it from configuration parameters if set at runtime:

rabbitmq-diagnostics environment | grep -i leader_locator

Count how many leaders sit on each node to quantify the skew:

rabbitmqctl list_queues name type leader | \
  awk 'NR>1 {print $3}' | sort | uniq -c

     41 rabbit@node1
      0 rabbit@node2
      0 rabbit@node3

Inspect a single quorum queue’s membership and current leader:

rabbitmq-queues quorum_status orders

Step-by-Step Resolution

1. Set the cluster-wide default locator to balanced so all new queues spread their leaders. Put this in rabbitmq.conf:

# /etc/rabbitmq/rabbitmq.conf
queue_leader_locator = balanced

2. Reload configuration (or restart nodes one at a time) so the setting takes effect for future declarations:

rabbitmqctl eval 'application:set_env(rabbit, queue_leader_locator, balanced).'

3. Rebalance the leaders of existing quorum and stream queues without recreating them:

rabbitmq-queues rebalance all

You can target a type explicitly:

rabbitmq-queues rebalance quorum
rabbitmq-queues rebalance stream

4. For a single stuck queue, transfer leadership to a specific node:

rabbitmq-queues transfer_leadership orders --node rabbit@node2

5. Remove any per-queue override that forces client-local. If applications declare with x-queue-leader-locator, drop that argument so queues honor the cluster default:

# remove from declare arguments:
x-queue-leader-locator: "client-local"

6. Verify the spread evened out:

rabbitmqctl list_queues name leader | awk 'NR>1 {print $2}' | sort | uniq -c

     14 rabbit@node1
     14 rabbit@node2
     13 rabbit@node3

Prevention

• Set queue_leader_locator = balanced in every node’s rabbitmq.conf from day one so leader placement is even by default.
• Spread client connections across all cluster nodes rather than pinning them to a single endpoint.
• After any rolling restart or node replacement, run rabbitmq-queues rebalance all as a standard post-maintenance step.
• Alert on per-node leader counts (via the Prometheus plugin) so a developing hot spot is caught early.
• Avoid hardcoding x-queue-leader-locator: client-local in application declare calls; validate declarations with prompts from the DevOps prompt library.

Related Errors

• quorum_status shows a node listed but not in sync — a follower still catching up, distinct from leader placement.
• khepri/ra election churn — repeated leader elections rather than static skew.
• Stream leader on a node without local disk headroom — placement succeeded but capacity is the real limit.
• rebalance reports queues it cannot move — minority of members offline, blocking safe transfer.

Frequently Asked Questions

What does queue_leader_locator = balanced actually do? When a new quorum queue or stream is declared, RabbitMQ places its leader on the node hosting the fewest queues, spreading load instead of following the client’s connection.

Does changing the locator move existing leaders? No. The setting only affects newly declared queues; run rabbitmq-queues rebalance all to redistribute leaders that already exist.

Is rebalancing safe on a live cluster? Yes — rabbitmq-queues rebalance transfers leadership gracefully between in-sync members without dropping messages, though clients briefly reconnect to the new leader.

Why did all leaders end up on one node after a restart? Queues failed over to the surviving nodes and leadership was never moved back; a post-restart rebalance corrects it. For more messaging fixes, see the RabbitMQ guides.