🚨 RabbitMQ Error Guide: 'UNEXPECTED_FRAME' — Fix AMQP Protocol Violations

Overview

The broker received an AMQP frame that violated the expected frame sequence on a channel and closed the connection with a fatal protocol error:

UNEXPECTED_FRAME - expected content header for class 60, got non content header frame instead

AMQP publishing is a strict sequence: a method frame (basic.publish) must be followed by a content-header frame, then one or more body frames. If frames arrive out of that order or interleaved, the broker cannot parse the stream and terminates the connection — this is almost always a client-side bug, not a broker fault.

Symptoms

• Connections close abruptly with UNEXPECTED_FRAME under concurrency or load.
• The error is intermittent and correlates with multi-threaded publishing.
• A specific client library or version triggers it; others on the same broker are fine.
• The broker log records a connection ... closed at the framing layer with the offending class.
• Consumers and publishers sharing one channel see corrupted delivery/publish behavior.

Common Root Causes

• Channel shared across threads — AMQP channels are not thread-safe; two threads writing to one channel interleave frames and corrupt the sequence.
• Interleaved publishes — an async client sending overlapping basic.publish sequences on the same channel.
• Buggy or outdated client — a library with a framing bug emits frames in the wrong order.
• Manual/custom AMQP implementation — hand-rolled protocol code that mis-orders header and body frames.
• Corrupted stream — a proxy or middlebox mangling the TCP byte stream between client and broker.
• Mismatched frame_max — a client sending a body frame larger than the negotiated frame_max.

Diagnostic Workflow

Find the connection and the offending frame in the broker log:

grep -iE 'UNEXPECTED_FRAME|expected content header|closing AMQP connection' \
  /var/log/rabbitmq/rabbit@$(hostname -s).log | tail

Identify the client (IP, user, library) behind the closed connection so you can pin the bug to a service:

rabbitmqctl list_connections name peer_host user protocol client_properties

Check the negotiated frame_max in case an oversized body frame is the trigger:

rabbitmqctl environment | grep -i frame_max
rabbitmqctl list_connections name frame_max

Confirm the client’s concurrency model — a single channel used by multiple threads/goroutines is the prime suspect. Reproduce by loading the suspect publisher and watching for the error to recur.

Example Root Cause Analysis

A Go service began dropping connections with UNEXPECTED_FRAME - expected content header for class 60 under load. It was fine at low volume.

rabbitmqctl list_connections name peer_host client_properties traced the closures to one service. Reviewing its code, a single AMQP channel was shared across multiple goroutines that each called publish concurrently. Because a channel is a single ordered frame stream, two goroutines interleaved their basic.publish + content-header sequences, so the broker saw a non-header frame where a content header belonged.

The fix was one channel per goroutine (channels are cheap; connections are the expensive resource):

// before: shared ch across goroutines  -> UNEXPECTED_FRAME
// after:  each goroutine opens its own channel on a shared connection

After the change the error disappeared under the same load. Root cause: violating the AMQP one-writer-per-channel rule.

Prevention Best Practices

• Never share a channel across threads/goroutines — use one channel per concurrent publisher/consumer.
• Reuse a small number of long-lived connections; open cheap channels per worker rather than sharing.
• Keep client libraries up to date to pick up framing-bug fixes.
• Avoid hand-rolled AMQP framing; use a maintained client that guarantees correct frame order.
• Ensure any proxy/LB in front of the broker is TCP-transparent and doesn’t alter the byte stream.

Quick Command Reference

grep 'UNEXPECTED_FRAME' /var/log/rabbitmq/*.log      # confirm + offending class
rabbitmqctl list_connections name peer_host user client_properties  # pin the client
rabbitmqctl list_connections name frame_max          # negotiated frame size
rabbitmqctl list_channels connection number          # channel usage per connection

Conclusion

UNEXPECTED_FRAME is an AMQP protocol violation: frames arrived out of the required method-header-body order on a channel, so the broker closed the connection. The overwhelming cause is a channel shared across threads or interleaved publishes — give every concurrent worker its own channel on a shared connection. Trace the connection to its client with list_connections, upgrade buggy libraries, and rule out a stream-mangling proxy. Correct per-channel concurrency eliminates the error.