🚨 RabbitMQ Error Guide: 'file descriptor limit alarm set' FD Exhaustion

Exact Error Message

When RabbitMQ runs out of available file descriptors it raises a resource alarm and writes a warning to its log. The exact wording depends on your version, but it looks like this:

=WARNING REPORT==== 24-Jun-2026::14:07:51.382611 ===
file descriptor limit alarm set.

***  ***  ***  ***  ***  ***  ***  ***  ***  ***  ***  ***  ***
*** New connections will not be accepted until this alarm clears ***
*** Publishers will be blocked until this alarm clears ***
***  ***  ***  ***  ***  ***  ***  ***  ***  ***  ***  ***  ***

=INFO REPORT==== 24-Jun-2026::14:07:51.383905 ===
File descriptors: total used 1024, total limit 1024
Sockets used: 980, socket limit 829

You may also see the alarm reflected in the management UI as a red banner reading “file descriptor alarm” and in rabbitmq-diagnostics alarms output. Clients usually report symptoms first: connection refused, hung publishes, or timeouts, because the broker has stopped accepting new TCP connections.

What the Error Means

Every TCP connection, channel socket, open queue index segment, and message store file in RabbitMQ consumes an operating-system file descriptor (FD). The Erlang VM that RabbitMQ runs on is given a hard ceiling on how many FDs the process may hold, derived from the OS ulimit -n (or systemd’s LimitNOFILE).

RabbitMQ does not wait until it hits that ceiling. It computes a high watermark — by default file_handles_high_watermark is 0.95 of the FD limit — and reserves a portion of FDs purely for sockets (the “socket limit” you saw above, which is 0.9 of the total by default). When usage crosses the watermark, RabbitMQ sets the file descriptor limit alarm as a protective measure. This is the same alarm machinery used for the memory and disk resource alarms, but triggered by FD exhaustion instead.

While the alarm is set, the broker refuses new connections and blocks publishing channels (consumers continue to drain queues). The goal is to give the broker headroom to flush data to disk safely rather than crashing with an emfile (too many open files) error. The alarm clears automatically once FD usage drops back below the watermark.

Common Causes

• Low ulimit -n. The default soft limit on many Linux distributions is 1024. A busy broker exhausts that almost instantly.
• systemd LimitNOFILE not set. When RabbitMQ runs under systemd, shell ulimit settings and /etc/security/limits.conf are ignored. Only the unit’s LimitNOFILE applies, and if it is unset the process inherits a low default.
• Too many connections or channels. Each connection plus its channels holds sockets. Applications that open one connection per request, or thousands of short-lived workers, drive FD count up fast.
• Socket leaks from churning clients. Clients that crash, disconnect uncleanly, or never close connections leave sockets in CLOSE_WAIT until the OS reaps them, inflating the count.
• Large numbers of queues. Queue index and message store files each consume FDs, so tens of thousands of queues add steady baseline pressure.
• Aggressive file_handles_high_watermark. A misconfigured watermark set too low will trip the alarm even when plenty of FDs remain.

How to Reproduce the Error

In a non-production lab you can trigger the alarm safely by starting the broker with a deliberately tiny FD limit and then opening more connections than it allows.

# Start a throwaway broker with a low FD ceiling (lab only)
ulimit -n 256
rabbitmq-server &

# Confirm the low ceiling the broker sees
rabbitmq-diagnostics status | grep -A3 "File Descriptors"

Then open a few hundred idle connections from a load tool such as perf-test or a small script that creates connections in a loop without closing them. As open sockets approach the socket limit, RabbitMQ logs file descriptor limit alarm set and new connection attempts begin to fail. Stopping the client and letting sockets close clears the alarm.

Diagnostic Commands

All of the commands below are read-only. Run them on the broker node.

Check current alarms and overall status:

rabbitmq-diagnostics alarms
rabbitmq-diagnostics status

Inspect the FD section of rabbitmqctl status directly:

rabbitmqctl status | grep -A6 -i "file descriptors"

File Descriptors

Total: 1024, limit: 1024
Sockets: 980, limit: 829

Count active connections and see how many sockets they hold:

rabbitmqctl list_connections name peer_host channels state | head -n 20
rabbitmqctl list_connections | wc -l

Confirm the OS-level limit the user sees versus what the running process actually has:

# Soft limit in the current shell
ulimit -n

# What systemd grants the unit
systemctl show rabbitmq-server -p LimitNOFILE

# The live limit on the actual beam.smp process (read-only)
prlimit --pid "$(pgrep -f beam.smp | head -n1)" --nofile

Look at OS socket state to spot leaks and CLOSE_WAIT build-up:

ss -s
ss -tan state close-wait | wc -l

Count open files held by the broker process as a cross-check:

lsof -p "$(pgrep -f beam.smp | head -n1)" | wc -l

If Sockets is near limit and ss -s shows a large socket count, you have a connection-volume or leak problem. If Total limit itself is 1024 or some other small number, the FD ceiling is too low and needs raising.

Step-by-Step Resolution

1. Confirm the alarm and its source. Run rabbitmq-diagnostics alarms. If it reports a file_descriptor_limit alarm, compare Total used against limit in rabbitmqctl status. This tells you whether the limit is too low or usage is abnormally high.

2. Relieve immediate pressure (clients). The fastest non-destructive relief is on the client side: stop runaway workers, fix code that opens a connection per request, and ensure clients close connections and channels. As idle sockets close, FD usage drops below the watermark and the alarm clears on its own.

3. Raise the systemd limit (the real fix on most hosts). Edit the unit override so the broker is granted enough FDs. A drop-in at /etc/systemd/system/rabbitmq-server.service.d/limits.conf with LimitNOFILE=300000 is typical for busy brokers. After adding it, reload the daemon and restart the service during a maintenance window:

systemctl daemon-reload
systemctl restart rabbitmq-server

4. Verify the new ceiling took effect. Re-run systemctl show rabbitmq-server -p LimitNOFILE and rabbitmqctl status | grep -A6 -i "file descriptors". The Total limit should now reflect the new value, and the Sockets limit should be roughly 90% of it.

5. Address leaks if usage is still climbing. If FD count keeps growing under steady load, you have a leak. Use ss -tan state close-wait | wc -l to confirm, then fix the offending client to close connections cleanly. Enable connection heartbeats so the broker can reap dead peers.

6. Tune the watermark only if justified. file_handles_high_watermark defaults to 0.95. Leave it unless you have a specific reason; raising the underlying limit is almost always the correct lever rather than loosening the watermark.

If you want this kind of triage automated against live broker output, our incident response assistant can correlate the alarm log with rabbitmqctl status and suggest the limit change.

Prevention and Best Practices

• Set LimitNOFILE explicitly in the systemd unit for every broker — never rely on the inherited default. 65536 is a sane minimum; 300000 for high-connection clusters.
• Use long-lived, pooled connections and reuse channels rather than opening one per operation.
• Enable heartbeats (the default 60s is fine) so half-open sockets are detected and closed.
• Monitor Sockets used versus limit and alert at 70% so you act before the alarm fires.
• Periodically audit connection count with rabbitmqctl list_connections and investigate unexpected growth.
• Keep queue counts reasonable; tens of thousands of queues quietly consume FDs.

Related Errors

The file descriptor alarm is one of three resource alarms RabbitMQ can raise. The memory resource alarm (vm_memory_high_watermark) and the disk resource alarm (disk_free_limit) block publishers in the same way but for different resources — covered in our separate memory/disk alarm guide. You may also encounter “too many channels” when an application opens excessive channels on one connection, and connection_closed_abruptly, which often accompanies FD pressure because clients are dropped when the broker stops accepting traffic. The latter two frequently appear together with the FD alarm when a churning client is the root cause.

Frequently Asked Questions

Does the file descriptor alarm stop consumers as well as publishers? No. Consumers keep draining existing queues. Only new connections and publishing channels are blocked, so backlogs can still be worked off while the alarm is active.

Why didn’t my /etc/security/limits.conf change take effect? Because RabbitMQ runs under systemd, which ignores limits.conf and ulimit. You must set LimitNOFILE in the systemd unit (a drop-in override) and run systemctl daemon-reload plus a restart.

How do I know whether to raise the limit or fix a leak? Compare Total used to limit. If you are near the limit but the limit is small (e.g. 1024), raise it. If the limit is already large and usage keeps climbing under steady load, you have a leak — check ss -tan state close-wait.

Will the alarm clear without restarting RabbitMQ? Yes. It clears automatically once FD usage falls below the high watermark, for example after misbehaving clients disconnect. A restart is only needed to apply a new LimitNOFILE.

What value should LimitNOFILE be? For most production brokers, 65536 is the floor and 300000 is common for clusters with many connections. Size it well above your peak Sockets used with headroom. For more RabbitMQ guidance see /categories/rabbitmq/.