🚨 RabbitMQ Error Guide: 'Error on AMQP connection' Reading the Connection Lifecycle Logs

Exact Error Message

The connection-lifecycle log family is the single most useful set of lines RabbitMQ emits. Every client connection produces an accepting line when it arrives, a closing line when it leaves cleanly, and an Error on AMQP connection line when something goes wrong. Learning to read the close reason in these lines short-circuits most connectivity debugging.

2026-06-29 14:02:11.482 [info] <0.18843.7> accepting AMQP connection <0.18843.7> (10.0.6.42:51544 -> 10.0.4.21:5672)
2026-06-29 14:02:11.501 [info] <0.18843.7> connection <0.18843.7> (10.0.6.42:51544 -> 10.0.4.21:5672): user 'orders-svc' authenticated and granted access to vhost 'prod'
2026-06-29 14:09:33.210 [warning] <0.18843.7> closing AMQP connection <0.18843.7> (10.0.6.42:51544 -> 10.0.4.21:5672, vhost: 'prod', user: 'orders-svc'):
client unexpectedly closed TCP connection
2026-06-29 14:11:02.778 [error] <0.19002.7> Error on AMQP connection <0.19002.7> (10.0.6.42:51790 -> 10.0.4.21:5672, state: starting):
{handshake_error,opening,{amqp_error,access_refused,"...",none}}

The pid in angle brackets (<0.18843.7>) ties every line for one connection together. The text after the colon on the closing or Error on line is the close reason — that is what you actually need to read.

What the Error Means

These three log shapes describe the lifecycle of a single TCP/AMQP connection:

• accepting AMQP connection — the TCP socket was accepted and an AMQP connection process (the pid) was spawned. The connection has not authenticated yet.
• closing AMQP connection — the connection is being torn down. The trailing reason explains why: a clean client close, a missed heartbeat, or the peer dropping TCP.
• Error on AMQP connection ... state: <X> — the connection failed during a protocol phase. The state: field (starting, securing, opening, running) tells you how far it got, and the Erlang tuple after it is the reason.

The key skill is mapping the reason text to a category: a clean closing with “client unexpectedly closed TCP connection” is a client/network issue, while {handshake_error, opening, ...} is an auth or vhost problem caught during the AMQP handshake.

Common Causes

1. Client closed without sending Connection.Close

A worker that crashes, is OOM-killed, or has its container stopped drops the TCP socket without the AMQP close handshake. RabbitMQ logs closing AMQP connection ... client unexpectedly closed TCP connection.

2. Handshake error during opening (auth / vhost)

{handshake_error, opening, {amqp_error, access_refused, ...}} or {handshake_error, starting, ...} means the connection reached the broker but failed authentication, authorization, or vhost selection during the protocol handshake.

3. Missed heartbeats

If the negotiated heartbeat interval elapses with no frames, the connection closes with missed heartbeats from client, timeout: 60s. This signals a stalled client or a network path that silently drops idle traffic.

4. Frame / protocol errors

A non-AMQP client (an HTTP health check, a port scanner, or a TLS client hitting the plaintext port) produces {bad_header, ...} or {refused, ...} errors during starting.

5. Server-initiated close

Resource alarms (memory/disk), rabbitmqctl close_connection, or a connection.blocked followed by a forced close appear as broker-side closing lines with a reason such as broker forced connection closure with reason 'shutdown'.

How to Reproduce the Error

Generate a clean lifecycle pair, then an abrupt close, against a test vhost:

# Clean open + close: a healthy short-lived connection
rabbitmqadmin -u guest -p guest declare queue name=lifecycle-test
# Abrupt close: open a raw socket, then kill it without an AMQP close
python3 - <<'PY'
import socket, time
s = socket.create_connection(("10.0.4.21", 5672))
s.sendall(b"AMQP\x00\x00\x09\x01")   # send the protocol header only
time.sleep(1)
s.close()                            # drop TCP without finishing the handshake
PY

The raw-socket snippet produces an accepting line followed by an Error on AMQP connection ... state: starting because the handshake never completes.

Diagnostic Commands

# Tail the broker log and isolate the lifecycle family
sudo journalctl -u rabbitmq-server -f | grep -E 'accepting|closing|Error on AMQP'

# Follow one connection by its pid across all its log lines
sudo journalctl -u rabbitmq-server --no-pager | grep '<0.18843.7>'

# List live connections with their state and peer
rabbitmqctl list_connections name peer_host peer_port state user vhost protocol

# Show negotiated timeouts and frame settings per connection
rabbitmqctl list_connections name timeout frame_max channel_max client_properties

# Confirm which listeners and protocols are active
rabbitmq-diagnostics listeners
sudo ss -tnp | grep -E ':5672|:5671'

Step-by-Step Resolution

Step 1: Find the connection by pid

Every lifecycle line for one connection shares the pid. Grep the log for that pid to see the full story — when it was accepted, whether it authenticated, and the exact close reason.

sudo journalctl -u rabbitmq-server --no-pager | grep '<0.19002.7>'

Step 2: Classify the close reason

Read the text after the colon and map it:

• client unexpectedly closed TCP connection → client crash or network drop (transport).
• {handshake_error, opening|starting, ...} → auth/authz/vhost (handle as an ACCESS_REFUSED).
• missed heartbeats from client → stalled client or idle-killing middlebox.
• connection_closed_abruptly / {bad_header, ...} → wrong protocol/port or proxy mangling frames.

Step 3: Correlate with the client

Match the peer host:port in the log to the client instance. Check that client’s own logs at the same timestamp — a clean broker-side closing often pairs with an OOM kill or deploy event on the client.

Step 4: Act on the category

For transport drops, investigate the client and any LB idle timeout. For handshake errors, fix credentials/permissions. For missed heartbeats, lower the heartbeat interval and ensure TCP keepalives traverse the path. For protocol errors, point the client at the correct port and protocol.

Step 5: Confirm a clean cycle

After fixing, you should see a matching accepting and a graceful closing ... connection closed cleanly (or no close at all for long-lived connections).

Prevention and Best Practices

• Centralize broker logs and build a saved query for the accepting|closing|Error on AMQP family so close reasons are searchable by pid and peer.
• Always close connections gracefully in client code (connection.close() on shutdown) so you can distinguish real failures from normal lifecycle noise.
• Set a sane heartbeat (e.g., 30-60s) on both client and server and verify it survives any proxy or NAT in the path.
• Alert on a rising rate of Error on AMQP connection and client unexpectedly closed lines — a spike usually precedes a client-fleet incident.
• Keep the AMQP and AMQPS ports clearly separated and ensure health checks never probe the AMQP port with non-AMQP traffic.
• For fast triage, the free incident assistant can turn a raw lifecycle log block into a likely cause.

Related Errors

• Connection refused (ECONNREFUSED) — fails before any accepting line is logged; the broker never sees the connection.
• missed heartbeats, timeout — a specific closing reason in this family with its own remediation.
• connection_closed_abruptly — the abrupt-close variant of the closing line.
• ACCESS_REFUSED - Login was refused — the authentication-failure form of {handshake_error, starting, ...}.

For more, see the RabbitMQ guides.

Frequently Asked Questions

What does the state: field in an Error on AMQP connection line mean? It is the protocol phase the connection reached before failing: starting (before/at authentication), securing (SASL exchange), opening (vhost access), and running (fully open). It tells you whether the failure was transport, auth, vhost, or runtime.

Is a closing AMQP connection line always a problem? No. A graceful application shutdown logs closing ... connection closed cleanly at info level, which is normal. Only reasons like “client unexpectedly closed TCP connection,” missed heartbeats, or broker-forced closures indicate trouble.

How do I tie a log line back to a specific client? Use the peer host:port shown in the line and the connection pid. rabbitmqctl list_connections name peer_host peer_port maps live connections, and grepping the log by pid reconstructs the full lifecycle of one connection.

Why do I see accepting lines from IPs I don’t recognize? Those are usually load-balancer health checks, port scanners, or monitoring probes opening and dropping TCP. If they never authenticate and close immediately, exclude that source or point health checks at a proper endpoint.

Can I reduce the volume of these log lines? You can raise the connection log category level, but it is better to keep them at info/warning and filter downstream — these lines are your primary signal for connection health.