🚨 RabbitMQ Error Guide: 'CONNECTION_FORCED' — Why the Broker Closed Your Connection

Overview

CONNECTION_FORCED is the broker telling a client “I am closing this connection on purpose.” It is an AMQP 0-9-1 connection-level exception (reply code 320) that RabbitMQ sends when the broker — not the network and not the client — decides to terminate the connection. Because it’s an orderly close initiated by the server, the reason string almost always tells you exactly why.

You will see it in the client:

pika.exceptions.ConnectionClosedByBroker: (320, "CONNECTION_FORCED -
broker forced connection closure with reason 'shutdown'")

The reason in the quotes is the whole story. Common ones:

CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
CONNECTION_FORCED - Closed via management plugin
CONNECTION_FORCED - broker forced connection closure with reason 'vhost deleted'
CONNECTION_FORCED - time to live exceeded

Unlike connection reset by peer (a network drop) or missed heartbeats (the client went silent), CONNECTION_FORCED is a deliberate, clean shutdown by the broker. The right response is almost never to “fix the network” — it’s to understand which deliberate action closed the connection and make sure your client reconnects gracefully.

Symptoms

• Clients disconnect with reply code 320 and a human-readable reason string.
• Disconnects cluster around a known event: a node restart, an upgrade, a management action, or a vhost change.
• The broker log records an orderly closing AMQP connection rather than an error/reset.
• Clients with auto-recovery reconnect seconds later; clients without it stay down.

sudo grep -i 'connection_forced\|closing AMQP connection' \
  /var/log/rabbitmq/rabbit@$(hostname -s).log | tail -5

=INFO REPORT==== closing AMQP connection <0.1720.0> (10.0.5.31:52310 -> 10.0.4.21:5672 - conn123): reason 'shutdown'

Common Root Causes

1. Node shutdown, restart, or rolling upgrade

When a node stops (maintenance, upgrade, restart), it forcibly closes every connection it owns with reason 'shutdown'. Expected — but clients must reconnect, ideally to another node.

sudo journalctl -u rabbitmq-server --since '30 min ago' | grep -i 'stopping\|stopped\|shutdown'

Stopping RabbitMQ Application...

If the disconnects line up with a node stop, this is planned (or unplanned) node lifecycle, not a client bug.

2. A connection closed manually via the management UI or API

An operator (or an automation) closed the connection through the management plugin, producing reason Closed via management plugin.

sudo grep 'Closed via management plugin' /var/log/rabbitmq/rabbit@$(hostname -s).log | tail

Someone — or a script — deliberately terminated that connection. Find out who and why before assuming a defect.

3. A vhost or user was deleted or its permissions revoked

Deleting a vhost, or removing a user’s access, forcibly closes every connection using it, often with reason 'vhost deleted' or an access revocation.

rabbitmqctl list_vhosts
rabbitmqctl list_permissions -p <VHOST>

If a vhost the client used is gone, that’s the cause — a config or provisioning change pulled it out from under live connections.

4. An idle-connection or maintenance reaper

Some deployments run automation that closes idle or old connections (via the management API), or a blue/green cutover that drains a node. These arrive as CONNECTION_FORCED with a custom reason.

rabbitmqctl list_connections name user vhost state channels | sort

Cross-reference the closed connections against any draining/reaper automation you run.

5. The client never recovers because auto-recovery is off

Not a broker cause, but the reason CONNECTION_FORCED becomes an outage: many client libraries disable automatic recovery by default, so a perfectly normal 'shutdown' close leaves the app permanently disconnected.

rabbitmqctl list_connections name client_properties | grep <CLIENT_IP>

10.0.5.31:52310  [{"product","pika"},{"version","1.3.2"},...]

Some libraries (e.g., pika) require you to implement reconnect yourself; others (Java, .NET, Bunny) have opt-in automatic recovery.

Diagnostic Workflow

Step 1: Read the reason string — it is the diagnosis

sudo grep -i 'connection_forced\|closing AMQP connection' \
  /var/log/rabbitmq/rabbit@$(hostname -s).log | tail -20

'shutdown' → node lifecycle. Closed via management plugin → a manual/automated close. 'vhost deleted' → a provisioning change. Match the reason to the cause; don’t guess.

Step 2: Correlate the timing with broker events

sudo journalctl -u rabbitmq-server --since '1 hour ago' \
  | grep -iE 'stopping|stopped|starting|shutdown'

If closures coincide with a node stop/start, the connection close was expected and the question shifts to client recovery.

Step 3: Check whether it was a management-initiated close

sudo grep 'Closed via management plugin' \
  /var/log/rabbitmq/rabbit@$(hostname -s).log | tail

A management-plugin close is a human or a script — audit who has management access and what automation runs against the API.

Step 4: Verify the vhost and permissions still exist

rabbitmqctl list_vhosts
rabbitmqctl list_permissions -p <VHOST>

A missing vhost or revoked permission explains a CONNECTION_FORCED and points at a provisioning/GitOps change rather than the broker.

Step 5: Confirm the client reconnects

The broker did its job; the outage, if any, is on the client. Verify auto-recovery is enabled or that the app catches the 320 and reconnects with backoff.

watch -n 5 "rabbitmqctl list_connections name user | grep <CLIENT_IP> | wc -l"

A connection reappearing within seconds confirms recovery is working.

Example Root Cause Analysis

During a routine rolling upgrade, an on-call engineer gets paged: a batch worker fleet stops consuming and never recovers. The client logs show:

pika.exceptions.ConnectionClosedByBroker: (320, "CONNECTION_FORCED -
broker forced connection closure with reason 'shutdown'")

The reason is 'shutdown', and the timing matches the upgrade exactly:

sudo journalctl -u rabbitmq-server --since '20 min ago' | grep -i 'stopping\|stopped'

Stopping RabbitMQ Application...
RabbitMQ Application stopped.

So the broker behaved correctly — node rabbit@node2 was drained for the upgrade and forcibly closed its connections with 'shutdown', exactly as designed. The other cluster nodes stayed up the whole time. The real defect is on the client: the pika-based workers open a single blocking connection at startup and have no reconnect logic, so a normal node close permanently disconnects them.

The fix is client-side resilience — reconnect with exponential backoff and re-declare consumers on close:

# pseudocode: on ConnectionClosedByBroker(320), reconnect with backoff,
# then re-open channel, re-set QoS, and re-register the consumer
while True:
    try: run_consumer()
    except ConnectionClosedByBroker: sleep(backoff()); continue

After deploying reconnect logic (and pointing clients at all cluster nodes so they land on a surviving one), the next rolling upgrade causes a brief blip instead of an outage. CONNECTION_FORCED - 'shutdown' was never the problem — the missing reconnect was.

Prevention Best Practices

• Read the reason string first — it distinguishes a planned node shutdown from a manual close or a vhost change, and dictates whether anything is actually wrong.
• Build client auto-recovery: enable the library’s automatic recovery (Java/.NET/Bunny) or implement reconnect-with-backoff and consumer re-registration (pika and other manual libraries) so a normal broker close is a blip, not an outage.
• Connect clients to all cluster nodes (or a load balancer) so a 'shutdown' on one node lets clients reconnect to a surviving one.
• Treat vhost/user deletion as breaking — coordinate provisioning changes, since deleting a vhost forcibly closes every live connection using it.
• Audit management-plugin access and any connection-reaper automation so Closed via management plugin closures are accountable and expected.
• Alert on connection reconnect storms rather than individual 320s; a healthy fleet reconnects, so the signal is connections that stay down. The free incident assistant can correlate the closure reason with node-lifecycle events to confirm a close was planned. More in the RabbitMQ guides.

Quick Command Reference

# Read connection-forced closures and their reason strings
sudo grep -i 'connection_forced\|closing AMQP connection' \
  /var/log/rabbitmq/rabbit@$(hostname -s).log | tail -20

# Correlate with node lifecycle events
sudo journalctl -u rabbitmq-server --since '1 hour ago' \
  | grep -iE 'stopping|stopped|starting|shutdown'

# Detect management-plugin-initiated closes
sudo grep 'Closed via management plugin' /var/log/rabbitmq/rabbit@$(hostname -s).log | tail

# Verify vhost and permissions still exist
rabbitmqctl list_vhosts
rabbitmqctl list_permissions -p <VHOST>

# Watch a client reconnect after the close
watch -n 5 "rabbitmqctl list_connections name user | grep <CLIENT_IP> | wc -l"

Conclusion

CONNECTION_FORCED (reply code 320) means the broker deliberately closed the connection and told you why in the reason string — it is an orderly server-initiated close, not a network failure. The usual reasons:

1. Node shutdown, restart, or rolling upgrade ('shutdown').
2. A manual or automated close via the management plugin.
3. A deleted vhost or revoked permissions.
4. An idle-connection reaper or node-draining automation.
5. A client with no auto-recovery, which turns a normal close into an outage.

Read the reason string first — it is the diagnosis. Most of the time the broker did exactly what it should, and the fix is client-side: reconnect with backoff, re-register consumers, and connect to all cluster nodes so a planned close is a momentary blip instead of a page.