🐰 RabbitMQ Error: Classic Mirrored Queue Synchronisation Stuck / Unsynchronised Mirrors

Exact Error Message

A classic mirrored queue whose mirrors have not caught up shows fewer synchronised mirrors than total mirrors, and manual/automatic sync appears to hang:

$ rabbitmqctl list_queues name slave_pids synchronised_slave_pids messages
Listing queues for vhost / ...
name        slave_pids                    synchronised_slave_pids   messages
orders      [<rabbit@mq-02.3.1180.0>,      []                        184203
             <rabbit@mq-03.3.1190.0>]

2026-07-17 13:05:22.774 [warning] <0.1180.0> Mirrored queue 'orders' in vhost '/':
    mirror <rabbit@mq-02> is not synchronised; messages will not be available on
    this mirror if the master fails

Triggering a sync and watching it not progress is the classic stuck symptom:

$ rabbitmqctl sync_queue orders
Synchronising queue 'orders' in vhost '/' ...
# ... no completion, synchronised_slave_pids stays []

What It Means

A classic mirrored (HA) queue has one master replica and one or more mirrors (historically “slaves”) on other nodes, configured by an ha-mode policy. A mirror is only useful if it is synchronised — holding the same messages as the master. synchronised_slave_pids lists the mirrors that are fully caught up; when it is empty (or shorter than slave_pids), those mirrors are unsynchronised and would lose data if the master failed.

Synchronisation copies the queue’s existing messages to the mirror. With ha-sync-mode: manual (the default), a new or restarted mirror does not auto-sync — it only receives new messages, so a deep queue can sit unsynchronised indefinitely until you run sync_queue. A sync that appears stuck is usually a very large queue copying slowly, or a queue so busy that it never reaches a sync point.

Note: classic mirrored queues are deprecated and removed in RabbitMQ 4.0. The durable fix is migrating to quorum queues.

Common Causes

• ha-sync-mode is manual, so a restarted or newly added mirror never automatically catches up.
• The queue holds a very large backlog, so synchronisation legitimately takes a long time and looks stuck.
• The queue is under constant high publish load, so it never reaches a quiescent point to complete a sync.
• A mirror node was down or slow, leaving it behind after it returned.
• Network or disk pressure between master and mirror throttling the copy.
• A recent failover promoted an unsynchronised mirror, which is exactly the data-loss risk the warning describes.

Diagnostic Commands

See which mirrors are synchronised versus merely present:

rabbitmqctl list_queues name slave_pids synchronised_slave_pids messages

Check the policy driving mirroring and its sync mode:

rabbitmqctl list_policies

Watch sync progress and any mirror warnings in the log:

sudo journalctl -u rabbitmq-server -f | grep -iE 'mirror|synchronis'

Confirm cluster members are all up (an unsynchronised mirror is often a recently-returned node):

rabbitmqctl cluster_status

List queues that currently have unsynchronised mirrors across the vhost:

rabbitmqctl list_queues name synchronised_slave_pids slave_pids \
  | awk 'NF && $2=="[]"'

Step-by-Step Resolution

1. Confirm the queue really is unsynchronised: synchronised_slave_pids shorter than slave_pids (often []). Note the message count — a huge queue explains a slow sync.

2. Trigger synchronisation explicitly and watch it progress:

rabbitmqctl sync_queue orders

Re-run list_queues ... synchronised_slave_pids every few seconds; the mirror PIDs should move into the synchronised list.

3. If the sync never completes because the queue is huge or constantly busy, sync during a quieter window, or drain/reduce the backlog first so the copy can finish.

4. If a sync is genuinely wedged and blocking, cancel it and retry cleanly:

rabbitmqctl cancel_sync_queue orders
rabbitmqctl sync_queue orders

5. Stop this recurring by switching to automatic sync so restarted mirrors catch up on their own:

rabbitmqctl set_policy ha-orders '^orders$' \
  '{"ha-mode":"exactly","ha-params":2,"ha-sync-mode":"automatic"}' \
  --apply-to queues

Be aware automatic sync briefly blocks the queue while copying — acceptable for most, risky for very deep queues.

6. For a permanent fix, migrate the queue to a quorum queue (Raft-replicated, no manual sync). This requires recreating the queue as x-queue-type: quorum and moving producers/consumers over:

rabbitmqctl list_queues name type | grep orders

7. Verify all mirrors are synchronised:

rabbitmqctl list_queues name slave_pids synchronised_slave_pids

synchronised_slave_pids should now match slave_pids for the queue.

Prevention

• Migrate replicated classic mirrored queues to quorum queues, which self-replicate via Raft and eliminate manual synchronisation entirely.
• If you must keep mirrored queues, set ha-sync-mode: automatic so restarted mirrors resync without operator action.
• Keep queues shallow; a large backlog turns every sync into a long, blocking copy.
• Alert on queues where synchronised_slave_pids is shorter than slave_pids, so an unsynchronised mirror is caught before a failover.
• Avoid promoting unsynchronised mirrors — a failover to a behind mirror silently drops messages.

For prompts that turn a list_queues sync report into a migration plan, see the RabbitMQ prompt library.

Related Errors

• Classic mirrored queue policy deprecated — the deprecation warning that precedes removal in 4.0.
• Node rabbit@host is down — a down mirror node is a common reason a mirror falls out of sync.
• Mnesia network partition — a partition can leave mirrors permanently unsynchronised on the minority side.

Frequently Asked Questions

Why won’t my new mirror synchronise on its own? The default ha-sync-mode is manual, so a newly added or restarted mirror only receives new messages and never copies the existing backlog until you run rabbitmqctl sync_queue. Switch to automatic to remove the manual step.

Is the sync actually stuck or just slow? Check the queue’s message count. Synchronisation copies every existing message to the mirror, so a queue with hundreds of thousands of messages can take a long time and only looks stuck. Watch synchronised_slave_pids for progress.

Is it safe to fail over to an unsynchronised mirror? No — an unsynchronised mirror does not hold all the master’s messages, so promoting it drops whatever it hadn’t received. That is precisely the risk the “not synchronised” warning describes.

Does ha-sync-mode: automatic have a downside? Yes: automatic sync briefly blocks the queue while it copies messages to a mirror. That’s fine for shallow queues but can cause a noticeable pause for very deep ones, so schedule accordingly.

What’s the long-term fix? Migrate to quorum queues. They replicate through Raft, need no manual synchronisation, and classic mirrored queues are removed in RabbitMQ 4.0. For more migration and clustering guidance, see the RabbitMQ guides.