🚨 RabbitMQ Error Guide: 'TLS handshake failed' Protocol and Cipher Negotiation Errors

Exact Error Message

A TLS handshake failure means the client and broker could not agree on the parameters of a secure session — protocol version, cipher suite, or how the connection was initiated — before any AMQP traffic flows. The connection dies during the SSL/TLS negotiation on port 5671.

2026-06-29 08:14:02.331 [error] <0.21560.2> TLS server: In state hello at tls_record.erl:561 generated SERVER ALERT: Fatal - Protocol Version
2026-06-29 08:14:02.331 [error] <0.21560.2> TLS server generated SERVER ALERT: Fatal - Handshake Failure - no_suitable_ciphers
2026-06-29 08:14:09.882 [error] <0.21577.2> Error on AMQP connection <0.21577.2> (10.0.6.42:52210 -> 10.0.4.21:5671, state: starting):
{ssl_upgrade_error,{tls_alert,{handshake_failure,"received CLIENT ALERT: Fatal - Handshake Failure"}}}

A plaintext client mistakenly hitting the TLS port shows the broker logging garbage as a bad TLS record:

2026-06-29 08:15:44.090 [error] <0.21601.2> TLS server: In state hello received a record with invalid record type, possibly a plaintext client on a TLS port

What the Error Means

TLS negotiation happens before AMQP. The client sends a ClientHello advertising the TLS versions and cipher suites it supports plus optional SNI; the broker picks a mutually supported version and cipher and returns a ServerHello. If there is no overlap, one side sends a fatal alert and the connection closes. RabbitMQ logs these as tls_alert / SERVER ALERT / CLIENT ALERT.

This guide covers negotiation failures — protocol version, cipher suite, SNI, or wrong-port — which are distinct from trust failures (certificate verification). Negotiation failures mean the two ends could not even establish an encrypted channel; trust failures mean they negotiated one but rejected each other’s certificate.

Common Causes

1. Protocol version mismatch

The broker requires TLS 1.2/1.3 while the client (or an old library) only offers TLS 1.0/1.1, or vice versa. Result: SERVER ALERT: Fatal - Protocol Version.

2. No common cipher suite

The broker’s configured ssl_options.ciphers list shares no entry with what the client offers — common after hardening cipher lists. Result: Handshake Failure / no_suitable_ciphers.

3. Plaintext client on the TLS port

A client using amqp:// (no TLS) connects to 5671, sending an AMQP header where a ClientHello is expected. The broker reports an invalid TLS record.

4. TLS client on the plaintext port

The reverse: an amqps:// client hits 5672, where the broker expects raw AMQP, and the handshake never completes.

5. SNI / server name mismatch with a fronting proxy

A TLS-terminating proxy or SNI-routing layer in front of RabbitMQ rejects or misroutes the ClientHello when the expected server name is absent or wrong.

6. Incomplete or wrong TLS configuration on the broker

Missing ssl_options keys, an unreadable key file, or a listener that is not actually doing TLS produces handshake errors on every attempt.

How to Reproduce the Error

Force a protocol/cipher mismatch and a wrong-port attempt with openssl:

# Offer only TLS 1.1 to a broker that requires 1.2+ -> protocol version alert
openssl s_client -connect 10.0.4.21:5671 -tls1_1 </dev/null

# Offer a single cipher the broker does not support -> handshake failure
openssl s_client -connect 10.0.4.21:5671 -tls1_2 -cipher 'DES-CBC3-SHA' </dev/null

# Plaintext AMQP client hitting the TLS port -> invalid TLS record on the broker
python3 - <<'PY'
import socket
s = socket.create_connection(("10.0.4.21", 5671))
s.sendall(b"AMQP\x00\x00\x09\x01")   # AMQP header, not a TLS ClientHello
print(s.recv(64)); s.close()
PY

Diagnostic Commands

# Negotiate and inspect what the broker actually offers
openssl s_client -connect 10.0.4.21:5671 -servername mq.internal </dev/null 2>&1 | \
  grep -E 'Protocol|Cipher|verify|alert'

# Which TLS versions does the broker enable?
rabbitmq-diagnostics tls_versions

# Which cipher suites are enabled, in OpenSSL naming?
rabbitmq-diagnostics cipher_suites --format openssl

# Confirm the TLS listener is up on 5671
rabbitmq-diagnostics listeners
sudo ss -ltnp | grep ':5671'

# Probe a specific TLS version handshake quickly
openssl s_client -connect 10.0.4.21:5671 -tls1_2 </dev/null 2>&1 | grep -E 'Protocol|Cipher'

# Watch the broker log for TLS alerts in real time
sudo journalctl -u rabbitmq-server -f | grep -E 'ALERT|tls_alert|ssl_upgrade_error|invalid record'

Step-by-Step Resolution

Step 1: Confirm it is negotiation, not trust

Protocol Version, no_suitable_ciphers, Handshake Failure, and invalid record type are negotiation problems. certificate verify failed / unknown ca are trust problems (a separate guide). This guide is for the former.

Step 2: Check protocol overlap

Compare rabbitmq-diagnostics tls_versions with the client’s supported versions. If the broker requires 1.2/1.3, upgrade the client TLS library or relax the broker only as a last resort.

Step 3: Check cipher overlap

rabbitmq-diagnostics cipher_suites --format openssl
openssl s_client -connect 10.0.4.21:5671 -tls1_2 </dev/null 2>&1 | grep Cipher

Ensure at least one cipher is common. Widen the broker’s ssl_options.ciphers to include a modern suite the client offers, or update the client.

Step 4: Verify port and scheme alignment

amqps:// must target the TLS listener (5671); amqp:// must target the plaintext listener (5672). An “invalid record type” log line is the signature of a plaintext client on 5671.

Step 5: Fix SNI / fronting proxy

If a proxy terminates or routes by SNI, pass the correct -servername and make the client send SNI. Confirm the proxy forwards ClientHello to the right backend.

Step 6: Validate broker TLS config

Ensure ssl_options references readable cert/key/CA files and that the listener is actually a TLS listener. Reload and re-test with openssl s_client.

Prevention and Best Practices

• Standardize on TLS 1.2+ across clients and broker, and track library versions so an old client cannot silently fall back to an unsupported protocol.
• Maintain a vetted cipher list that intersects with every client platform you support, and test it with openssl s_client after each change.
• Keep AMQP (5672) and AMQPS (5671) ports and client schemes clearly separated and documented to avoid wrong-port handshakes.
• When a proxy fronts RabbitMQ, configure and test SNI explicitly so ClientHello routing is deterministic.
• Add a synthetic TLS handshake check (openssl s_client) from a client-network host and alert on negotiation alerts before the app fleet is affected.
• For fast triage, the free incident assistant can turn a tls_alert log block into a likely negotiation cause.

Related Errors

• certificate verify failed / unknown ca — trust/validation failure after negotiation succeeds (separate guide).
• ECONNREFUSED on 5671 — no TLS listener at all, distinct from a handshake that starts and fails.
• Socket closed abruptly during SSL handshake — a peer dropping TCP mid-negotiation, often a wrong-port or proxy issue.
• Error on AMQP connection ... state: starting — the lifecycle line that wraps ssl_upgrade_error.

More in the RabbitMQ guides.

Frequently Asked Questions

How do I tell a negotiation failure from a certificate problem? Negotiation failures cite Protocol Version, no_suitable_ciphers, Handshake Failure, or invalid record type — the two ends could not agree on how to talk. Certificate problems cite certificate verify failed, unknown ca, or certificate expired — they agreed on a session but rejected a cert.

What causes “invalid record type” in the broker log? A non-TLS client connecting to the TLS port (5671). The broker expects a ClientHello but receives a raw AMQP header. Point the client at 5672 with amqp://, or switch it to amqps:// on 5671.

How do I see which ciphers and TLS versions the broker allows? Run rabbitmq-diagnostics tls_versions and rabbitmq-diagnostics cipher_suites --format openssl. Compare them with what the client offers, which openssl s_client -connect host:5671 will reveal.

The handshake works with openssl but fails from my app — why? Usually the app’s TLS library offers a narrower set of protocols/ciphers, or sends no SNI. Match the client’s TLS settings to what openssl s_client -servername proves works, especially the TLS version and SNI.

Should I lower the broker’s TLS requirements to fix this? Prefer upgrading the client. Weakening protocol or cipher requirements reduces security for every connection. Only relax the broker temporarily, and re-tighten once clients are upgraded.