🐰 RabbitMQ Error: vm_memory_high_watermark alarm set, connections blocked

Exact Error Message

When RabbitMQ crosses the memory high watermark it raises a resource alarm and blocks publishing connections:

2026-07-17 10:02:41.663 [warning] <0.324.0> memory resource limit alarm set on node rabbit@node-1.

**********************************************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************

Publishers that opened the connection see a connection.blocked notification, and any new publish stalls:

connection.blocked: reason="low on memory"

Once memory drops back below the threshold the broker logs the clear:

2026-07-17 10:05:12.201 [warning] <0.324.0> memory resource limit alarm cleared on node rabbit@node-1

What It Means

RabbitMQ protects itself from running out of RAM by watching total memory use against a configurable fraction of installed memory — the vm_memory_high_watermark. The default is 0.4 (40% of system memory) or an absolute byte value. When the broker’s memory use exceeds that threshold, it raises a memory alarm and blocks every connection that is publishing.

Blocking is intentional flow control, not a crash. Consumers keep running so the broker can drain queues and reclaim memory; only publishers are paused. The connection stays open but publishes hang until the alarm clears. Any node in a cluster raising a memory (or disk) alarm blocks publishers cluster-wide.

Common Causes

• Queues have grown large because consumers are slow, down, or absent, so messages accumulate in RAM.
• Many messages sit unacknowledged (large prefetch with slow consumers) and cannot be released.
• The watermark is set too low for the workload, or the node simply has too little RAM.
• A memory leak or high connection/channel count inflates non-message memory.
• Classic queues holding everything in memory instead of using lazy/quorum queues that page to disk.

Diagnostic Commands

Confirm an alarm is active and see all alarms on the node:

rabbitmq-diagnostics alarms
rabbitmq-diagnostics list_node_alarms

Show the current watermark setting and memory usage breakdown:

rabbitmq-diagnostics memory_breakdown --unit mb
rabbitmqctl status | grep -A3 vm_memory

Find the queues holding the most memory:

rabbitmqctl list_queues name messages messages_ready messages_unacknowledged memory --vhost / \
  | sort -k6 -n -r | head -10

List blocked connections (state blocked/blocking):

rabbitmqctl list_connections name state user | grep -i block

Watch the log for the alarm set/clear transitions:

journalctl -u rabbitmq-server --since "15 min ago" | grep -i 'memory resource limit alarm'

Step-by-Step Resolution

1. Verify the alarm is memory, not disk. rabbitmq-diagnostics alarms distinguishes {resource_limit,memory,...} from {resource_limit,disk,...}. They have different fixes.

2. Find what is consuming memory with rabbitmq-diagnostics memory_breakdown and the per-queue memory column. Usually one or two deep queues dominate.

3. Drain the queues by fixing consumers. The fastest safe clear is to restart or scale consumers so messages_ready falls. Lower prefetch if messages_unacknowledged is bloating memory.

4. Temporarily raise the watermark to unblock publishers while you drain (use with care — do not exceed real RAM):

   rabbitmqctl set_vm_memory_high_watermark 0.6

   Or set an absolute value:

   rabbitmqctl set_vm_memory_high_watermark absolute "6GB"

5. Convert large classic queues to quorum or lazy behavior so messages page to disk instead of pinning RAM. Set via policy:

   rabbitmqctl set_policy lazy-events "^events" '{"queue-mode":"lazy"}' \
     --apply-to queues --vhost /

6. Make the watermark change durable by adding it to rabbitmq.conf so it survives restarts:

   vm_memory_high_watermark.relative = 0.6

7. Verify the alarm clears and publishers resume:

   rabbitmq-diagnostics alarms
   rabbitmqctl list_connections name state | grep -i running

Prevention

• Keep queues short: alert on messages_ready and per-queue memory well before the watermark.
• Prefer quorum queues (or lazy mode for classic queues) for large or bursty workloads so data pages to disk.
• Size prefetch so unacked messages do not accumulate in memory.
• Monitor rabbitmq-diagnostics memory_breakdown and set the watermark to match real RAM with headroom for the OS.
• Scale consumers to keep up with peak publish rate rather than relying on the alarm as backpressure. The prompt library has prompts for building memory-alarm runbooks and capacity checks.

Related Errors

• disk_free_limit alarm — the disk-space equivalent that also blocks publishers.
• connection.blocked / connection.unblocked — the client-side flow-control notifications tied to any resource alarm.
• partitions detected — a network partition, a separate cluster-health problem.
• reject-publish overflow nack — a single queue’s length limit, not a broker-wide memory alarm.

Frequently Asked Questions

Why are my publishers hung but consumers still work? A memory alarm blocks only publishing connections so the broker can drain queues and reclaim memory; consumers are left running on purpose.

Does the connection get dropped when the alarm fires? No — the connection stays open and receives a connection.blocked notification; publishes simply pause until the alarm clears.

Is raising vm_memory_high_watermark a real fix? Only as a temporary measure to unblock while you drain. The durable fix is shorter queues, quorum/lazy queues, and enough RAM.

Why does one node’s alarm block the whole cluster? RabbitMQ applies resource alarms cluster-wide, so any node over its watermark blocks publishers on every node until it recovers.

How do I make the watermark setting stick? Put vm_memory_high_watermark.relative (or .absolute) in rabbitmq.conf; rabbitmqctl set_vm_memory_high_watermark only lasts until restart. For more, see the RabbitMQ guides.