🐰 RabbitMQ Error: 'connection.blocked' — Publishers Blocked by a Resource Alarm

Exact Error Message

# In the node log when an alarm fires:
[warning] <0.567.0> memory resource limit alarm set on node rabbit@rabbit-01.
[warning] <0.567.0>
*** Publishers will be blocked until this alarm clears ***

# Client side (connection.blocked handler / hung publish):
Connection blocked: "low on memory"
Connection blocked: "low on disk"

Publishing clients don’t get an error so much as they stop: the TCP connection is flow-controlled and basic.publish calls hang until the broker sends connection.unblocked.

What It Means

RabbitMQ protects itself with resource alarms. When memory usage crosses vm_memory_high_watermark, or free disk falls below disk_free_limit, the broker raises an alarm and sends a connection.blocked notification to every connection that is publishing. Those connections are held in flow control — the broker stops reading from them — so producers block while consumers keep draining queues.

This is intentional back-pressure, not a crash. The idea is to stop new messages from making an out-of-memory or out-of-disk condition worse, giving consumers time to catch up and the alarm to clear. But if nothing drains the backlog, publishers can appear frozen indefinitely. The message string (low on memory or low on disk) tells you which alarm fired.

Common Causes

• Queues have grown large because consumers are slow, disconnected, or absent, driving memory up.
• Free disk on the RabbitMQ data volume dropped below disk_free_limit (default ~50MB or mem_relative).
• A publisher flood without matching consumer throughput.
• vm_memory_high_watermark set too low for the workload, or the container memory limit is smaller than the broker thinks.
• Large messages or many unacknowledged messages held in memory.
• A stuck/paused consumer holding a huge number of unacked messages that can’t be released.

Diagnostic Commands

Check whether an alarm is currently set and which resource triggered it:

rabbitmq-diagnostics alarms
rabbitmq-diagnostics status | grep -iE 'alarm|memory|disk'

Look at current memory usage against the high watermark:

rabbitmq-diagnostics memory_breakdown
rabbitmqctl eval 'rabbit_alarm:get_alarms().'

Check free disk on the data directory against the configured limit:

df -h /var/lib/rabbitmq
rabbitmq-diagnostics check_alarms

Find the queues driving the backlog:

rabbitmqctl list_queues name messages messages_unacknowledged memory \
  --vhost / | sort -k2 -n | tail

Step-by-Step Resolution

1. Identify which alarm is active. low on memory and low on disk need different fixes:

rabbitmq-diagnostics alarms

2. For a disk alarm, free space on the data volume immediately (rotate/remove logs, expand the disk). The alarm clears automatically once free space rises above disk_free_limit:

df -h /var/lib/rabbitmq
journalctl --vacuum-size=200M          # example: reclaim journal space

3. For a memory alarm, get consumers draining the backlog. Confirm consumers are attached and processing:

rabbitmqctl list_queues name messages consumers messages_unacknowledged \
  | sort -k2 -n | tail

4. If the watermark is genuinely too tight for the workload, raise it deliberately (as a fraction of RAM), then reload config. Do not set it so high the OS OOM-kills the node:

# /etc/rabbitmq/rabbitmq.conf
vm_memory_high_watermark.relative = 0.6
disk_free_limit.absolute = 2GB

rabbitmqctl eval 'rabbit_disk_monitor:set_disk_free_limit("2GB").'
rabbitmqctl eval 'vm_memory_monitor:set_vm_memory_high_watermark(0.6).'

5. In containers, make sure RabbitMQ sees the cgroup memory limit so its watermark math is correct; set RABBITMQ_MEMORY_... or use mem_relative against the real limit.

6. Confirm the alarm cleared and publishers resumed:

rabbitmq-diagnostics alarms

Node rabbit@rabbit-01 reported no alarms.

Prevention

• Handle connection.blocked/connection.unblocked in your client so publishers pause gracefully instead of appearing hung.
• Alert on rabbitmq_alarms_* (or the management API disk_free_alarm/mem_alarm) so you catch pressure before publishers block.
• Size vm_memory_high_watermark and disk_free_limit for the workload, and ensure container memory limits match what the broker assumes.
• Keep consumers healthy and scaled so queues drain faster than they fill; monitor queue depth trends.
• Set queue length limits (x-max-length) or use quorum queues with overflow policies so a runaway producer can’t exhaust memory.
• Provision generous disk on the data volume and monitor free space with headroom above the limit.

Related Errors

• vm_memory_high_watermark blocked connections — the memory-specific variant of this same alarm.
• x-overflow reject-publish / basic.nack — a per-queue overflow response, different from a node-wide resource alarm.
• disk_free_limit alarm — the disk-specific trigger for connection.blocked.
• PRECONDITION_FAILED on a queue — a queue-argument mismatch, unrelated to resource alarms.

Frequently Asked Questions

Is connection.blocked an error or normal behavior? It’s intentional back-pressure — the broker blocks publishers to protect itself from running out of memory or disk, and it unblocks them automatically when the alarm clears.

Why do my consumers keep working while publishers hang? Alarms only flow-control publishing connections; consumers keep draining so the backlog can shrink and the alarm can clear.

How do I stop publishers from appearing frozen? Register a connection.blocked/connection.unblocked handler in your client so it pauses and resumes explicitly instead of blocking on a hung basic.publish.

What clears the alarm? Freeing disk above disk_free_limit, or dropping memory below vm_memory_high_watermark (usually by draining queues), clears it automatically. For a workload-specific tuning plan, try the DevOps AI prompt library, or see more RabbitMQ guides.