🚨 RabbitMQ Error Guide: 'missed heartbeats' Heartbeat Timeout from Client

Overview

AMQP heartbeats are periodic frames the broker and client exchange to prove the connection is alive. If RabbitMQ sends a heartbeat and the client misses two consecutive heartbeat intervals, the broker concludes the client is dead and forcibly closes the connection. The negotiated heartbeat (commonly 60s) means a silent client gets dropped after about 2× that interval. This protects the broker from leaking resources on half-open connections, but it also means a client that blocks its own I/O — even while “healthy” — gets disconnected.

You will see it in the broker log:

=ERROR REPORT==== closing AMQP connection <0.1503.0> (10.0.5.31:51902 -> 10.0.4.21:5672):
missed heartbeats from client, timeout: 60s

And in the client, an abrupt connection drop:

ConnectionClosedByBroker: (320) "CONNECTION_FORCED -
broker forced connection closure with reason 'shutdown'"

or simply a reset socket. The cause is almost always that the client failed to send heartbeats in time — usually because its event loop was blocked, the network silently dropped the idle connection, or the heartbeat value is mismatched/too low.

Symptoms

• Connections drop after ~2× the heartbeat interval, often during idle or heavy CPU periods.
• Broker log shows missed heartbeats from client, timeout: 60s.
• Clients reconnect, run fine, then drop again on the same cadence.

sudo grep -i 'missed heartbeats' /var/log/rabbitmq/rabbit@$(hostname -s).log | tail -5

=ERROR REPORT==== closing AMQP connection (10.0.5.31:51902 -> 10.0.4.21:5672): missed heartbeats from client, timeout: 60s
=ERROR REPORT==== closing AMQP connection (10.0.5.31:51990 -> 10.0.4.21:5672): missed heartbeats from client, timeout: 60s

rabbitmqctl list_connections name timeout state | head

10.0.5.31:51902  60  running

The timeout column shows the negotiated heartbeat for each connection.

Common Root Causes

1. The client’s event loop is blocked

Single-threaded clients (Node.js, Python with a blocking handler) that do long synchronous work can’t send heartbeats on time.

rabbitmqctl list_connections name timeout recv_oct send_oct | grep <CLIENT_IP>

10.0.5.31:51902  60  18204411  4096

send_oct from the broker grows while the client sends almost nothing back — the client isn’t writing heartbeat frames because its loop is busy. Move heavy work off the I/O thread.

2. A firewall or load balancer idle-timeout kills the TCP connection

A NAT/firewall/LB that closes idle TCP flows shorter than the heartbeat interval silently severs the connection between heartbeats.

rabbitmqctl list_connections name timeout last_blocked_age | sort

10.0.5.31:51902  60  -

If a connection drops exactly at the firewall’s idle timeout (e.g., 30s) but the heartbeat is 60s, heartbeats never get a chance to keep it alive. Lower the heartbeat below the idle timeout.

3. Heartbeat negotiated too low

A very low heartbeat (e.g., 5–10s) is intolerant of brief GC pauses or scheduling delays and disconnects healthy clients.

rabbitmqctl list_connections name timeout | sort -k2 -n | head

10.0.5.31:52040  5  running

A 5s heartbeat means a ~10s stall drops the client — far too aggressive for most workloads. Raise it toward the default 60s.

4. Heartbeats disabled (timeout 0) and the connection silently dies

If heartbeats are turned off (heartbeat=0), nothing detects a dead path; the broker only notices when it next tries to write.

rabbitmqctl list_connections name timeout | awk '$2==0'

10.0.5.31:52111  0  running

timeout 0 means no heartbeats. Half-open connections linger and traffic stalls without a clean error. Enable heartbeats.

5. An overloaded consumer can’t service the socket

A consumer pegged at 100% CPU or swapping cannot read/write its socket promptly, so heartbeat frames back up.

rabbitmqctl list_connections name timeout channels recv_oct | grep <CLIENT_IP>

10.0.5.31:51902  60  50  21044112

A connection with many channels and high throughput on a starved host falls behind on heartbeats. Reduce prefetch/channels or scale the consumer.

6. Client/broker heartbeat mismatch from old libraries

Some older client libraries don’t honor the negotiated heartbeat or have buggy heartbeat threads, sending none.

rabbitmqctl list_connections name client_properties | grep <CLIENT_IP>

10.0.5.31:51902  [{"product","Bunny"},{"version","1.1.0"},...]

The client_properties reveal the library/version — a known-buggy heartbeat implementation is worth upgrading.

Diagnostic Workflow

Step 1: Confirm the broker is closing on missed heartbeats

sudo grep -i 'missed heartbeats from client' \
  /var/log/rabbitmq/rabbit@$(hostname -s).log | tail -10

This is unambiguous — the broker dropped the client because it didn’t hear heartbeats. Note the timeout: value.

Step 2: Read the negotiated heartbeat per connection

rabbitmqctl list_connections name timeout state recv_oct send_oct | sort

A timeout of 0 (disabled), a very low value, or a value larger than any intervening idle-timeout each point at a different fix.

Step 3: Check whether the client is blocked or just disconnected

rabbitmqctl list_connections name timeout recv_oct send_oct | grep <CLIENT_IP>

Broker send_oct rising while client recv_oct (from the broker’s view, client→broker) stays flat means the client isn’t writing — a blocked event loop, not a network drop.

Step 4: Test for an idle-timeout device in the path

# from the client host, hold an idle TCP connection and watch when it drops
nc <BROKER_HOST> 5672   # leave idle; if it resets well before 2x heartbeat, suspect a NAT/LB timeout

If the socket dies faster than the heartbeat interval, a firewall/LB is the culprit; set the heartbeat below that device’s idle timeout.

Step 5: Apply the fix and verify stable connections

Tune the heartbeat (raise if too aggressive, lower if below an idle timeout), move blocking work off the I/O thread, or scale the consumer. Then watch for recurrence.

watch -n 5 "sudo grep -c 'missed heartbeats from client' /var/log/rabbitmq/rabbit@$(hostname -s).log"

A flat count after the change confirms the fix.

Example Root Cause Analysis

A Node.js worker fleet drops connections roughly every two minutes; the broker log shows missed heartbeats from client, timeout: 60s for each. The network team confirms no firewall idle timeout shorter than 60s, ruling out a NAT drop.

Inspecting one connection while a worker processes a large batch:

rabbitmqctl list_connections name timeout recv_oct send_oct | grep 10.0.5.31:51902

10.0.5.31:51902  60  102881  4096

The broker has received almost nothing from the client (send_oct 4096) for over a minute while the worker is busy. The worker runs CPU-heavy message transformation synchronously on the same event loop that the amqp library uses to send heartbeats. During a long batch, the loop is blocked, no heartbeat frames go out, and after two missed 60s intervals the broker closes the connection.

Fix: move the heavy transformation off the event loop (a worker thread / setImmediate chunking) so the heartbeat timer fires on schedule, and lower prefetch so a single message can’t monopolize the loop:

channel.prefetch(20)   // was 500; smaller batches keep the loop responsive

After deploying, the missed heartbeats count stops climbing and connections stay up across batch processing. The disconnects were self-inflicted by a blocked event loop, not the network.

Prevention Best Practices

• Keep the AMQP client’s I/O loop free — never run long synchronous work on the thread that sends heartbeats; offload to worker threads or chunk the work.
• Set the heartbeat below any NAT/firewall/load-balancer idle timeout in the path (a 30s idle device wants a heartbeat well under 30s).
• Don’t disable heartbeats (heartbeat=0) in production; you lose detection of dead connections.
• Tune prefetch so a single message or batch can’t block the consumer long enough to miss heartbeats.
• Keep client libraries current — older versions have known heartbeat bugs.
• Alert on the rate of missed heartbeats log lines so a pattern surfaces before it becomes a reconnect storm. When it pages, the free incident assistant can correlate the broker log with the per-connection send_oct to flag a blocked client. More in the RabbitMQ guides.

Quick Command Reference

# Confirm missed-heartbeat closures
sudo grep -i 'missed heartbeats from client' /var/log/rabbitmq/rabbit@$(hostname -s).log | tail -10

# Negotiated heartbeat and traffic per connection
rabbitmqctl list_connections name timeout state recv_oct send_oct | sort

# Connections with heartbeats disabled or very low
rabbitmqctl list_connections name timeout | awk '$2==0 || $2<10'

# Client library/version
rabbitmqctl list_connections name client_properties | grep <CLIENT_IP>

# Watch for recurrence after a fix
sudo grep -c 'missed heartbeats from client' /var/log/rabbitmq/rabbit@$(hostname -s).log

Conclusion

A missed heartbeats from client closure means the broker stopped hearing the client’s heartbeat frames and dropped a presumed-dead connection. The usual root causes:

1. A blocked client event loop that can’t send heartbeats during heavy work.
2. A firewall/NAT/LB idle timeout shorter than the heartbeat interval.
3. A heartbeat negotiated too low to tolerate brief stalls.
4. Heartbeats disabled (timeout 0), hiding dead connections.
5. An overloaded consumer that can’t service its socket.

Confirm the closure in the broker log, read the per-connection timeout and traffic counters to tell a blocked client from a network drop, then tune the heartbeat or unblock the client’s I/O so frames flow on schedule.