🚨 RabbitMQ Error Guide: 'consumer cancelled' basic.cancel Notification

Exact Error Message

The broker pushes an unsolicited basic.cancel to the consumer (consumer cancel notification). Clients surface it differently:

# Java client
com.rabbitmq.client.Consumer#handleCancel(consumerTag="amq.ctag-7Hq3...")
 invoked: server cancelled consumer for queue 'payments.process'

# pika
ConsumerCancelled: (<Basic.Cancel consumer_tag='ctag-1.9af2...'>)

# broker log
operation basic.cancel due to queue 'payments.process' deletion on
 connection <0.901.0> (10.0.5.31:51780 -> 10.0.4.21:5672), channel 1

This is distinct from the consumer issuing a basic.cancel. Here the broker cancels the consumer and notifies the client, which must re-subscribe to keep consuming.

What the Error Means

A consumer cancel notification is RabbitMQ telling a client “your subscription to this queue is over.” RabbitMQ sends it whenever a queue a consumer is attached to disappears or becomes unavailable. By default, classic mirrored/older setups did not notify clients; modern clients negotiate the consumer_cancel_notify capability so the broker can proactively push basic.cancel instead of leaving the consumer silently dead.

The notification is informational, not an error in the protocol sense — the channel stays open. But operationally it is critical: a consumer that ignores handleCancel will stop receiving messages with no exception and no reconnect, and the queue silently backs up.

It is worth contrasting this with the two other ways consumption can end. A connection drop ends consumption via a connection/channel close, which raises a shutdown signal the client cannot miss. A basic.cancel you send is a deliberate unsubscribe. The server-initiated cancel sits between them: the transport is healthy and no exception fires, so only a registered callback tells your application that the subscription is gone. That gap is exactly why ignored cancels are such a common cause of “the queue stopped draining and nothing logged an error.”

Common Causes

• The queue was deleted. Someone (or a deploy/cleanup job) ran delete_queue, or an auto-delete queue dropped after its last consumer left and was recreated.
• The node hosting the queue went down. For a classic (non-replicated) queue, if its home node fails, the queue becomes unavailable and consumers are cancelled.
• A quorum/mirrored queue leader changed. On failover the consumer may be cancelled and must re-consume from the new leader.
• An exclusive queue’s owning connection closed. Exclusive queues are deleted when their declaring connection drops, cancelling any consumer.
• TTL or queue expiry (x-expires) removed the queue. An idle queue with x-expires set was auto-deleted while a consumer was attached.
• A policy or operator action purged/recreated the queue. Topology changes that delete and redeclare the queue cancel existing consumers.

How to Reproduce the Error

Start a consumer, then delete its queue from another session:

# session A: consume
basic.consume(queue='payments.process', no_ack=false)

# session B: delete the queue
queue.delete(queue='payments.process')

# session A receives:
handleCancel(consumerTag='amq.ctag-7Hq3...')   # server-initiated basic.cancel

Or set x-expires and let the queue idle out:

queue.declare(queue='temp.work', arguments={'x-expires': 60000})
# after 60s idle with a consumer attached, the queue expires and the consumer is cancelled

Diagnostic Commands

# Does the queue still exist? (absence explains the cancel)
rabbitmqctl list_queues name messages state | grep -i payments.process

# Where does the queue live, and is its node up?
rabbitmqctl list_queues name type node members | grep -i payments.process

# Cluster node status to spot a downed home node
rabbitmqctl cluster_status

# Current consumers per queue (zero after a cancel that was not re-subscribed)
rabbitmqctl list_consumers queue_name consumer_tag channel_pid ack_required | head -20

# Recent cancel / queue deletion events in the log
journalctl -u rabbitmq-server --since "30 min ago" | grep -iE 'basic.cancel|queue.*delet|down'

# Policies that might set x-expires / auto-delete behavior
rabbitmqctl list_policies | grep -iE 'expires|ha-|queue'

A queue that is simply gone from list_queues confirms deletion/expiry; a queue that exists but whose node appears in cluster_status as down confirms a host-node failure.

Step-by-Step Resolution

1. Confirm whether the queue still exists. Run rabbitmqctl list_queues. If the queue is absent, it was deleted or expired — the cancel was correct and the client must recreate/redeclare and re-consume.

2. If the queue exists, check its node. Use list_queues name node and cluster_status. A classic queue on a downed node is unavailable until that node returns. Move critical queues to quorum queues so they survive node loss.

3. Handle the cancel in client code. Implement handleCancel (Java) / on_consumer_cancelled (pika) to redeclare the queue (idempotently) and re-issue basic.consume. A consumer that ignores the notification is dead but looks alive.

4. Eliminate accidental deletion. Audit who or what deleted the queue: deploy scripts, cleanup cron, auto-delete/x-expires settings, or exclusive queues whose connection dropped. Remove auto-delete/x-expires from queues that must persist.

5. For exclusive queues, keep the owning connection alive. Exclusive queues die with their connection; do not use them for long-lived shared consumers.

6. Re-establish consumption and verify. After re-subscribing, confirm with rabbitmqctl list_consumers that the queue again has the expected consumer count and messages is draining.

A typical incident: a nightly cleanup job deleted “stale” temporary queues by name pattern, but one matching queue was a live work queue with attached consumers. Every consumer received a server basic.cancel, none of them implemented handleCancel, and the producers kept publishing into a freshly auto-declared queue with no readers. The backlog was only noticed hours later via a queue-depth alert. The fix had two parts: tighten the cleanup job’s pattern so it cannot match live queues, and add a handleCancel that redeclares and re-consumes. With both in place, even an accidental delete now self-heals within seconds instead of silently stalling the pipeline.

Prevention and Best Practices

• Always implement the consumer-cancel callback and re-subscribe; never assume a subscription is permanent.
• Use quorum queues for important workloads so a node failure triggers leader election instead of permanent unavailability.
• Avoid auto-delete and x-expires on queues that real consumers depend on long-term.
• Reserve exclusive queues for temporary, per-connection use only.
• Make queue and binding declaration idempotent so a re-subscribe can safely recreate topology.
• Alert on consumer count dropping to zero on critical queues, which catches an ignored cancel before the backlog grows.

Related Errors

• consumer timeout: the broker cancels/requeues when an ack is not delivered in time — a different trigger than queue deletion.
• quorum queue no leader elected: a related availability failure where the queue exists but has no leader to consume from.
• missed heartbeats / timeout: a dropped connection can also end consumption, but via connection close rather than basic.cancel.
• no route / unroutable message: publisher-side counterpart when a queue/binding is missing.

Frequently Asked Questions

Is a consumer cancel notification an error? Not at the protocol level — the channel stays open. Operationally it is critical because the consumer stops receiving messages until it re-subscribes.

Why did I not get a notification on an older setup? The broker only pushes basic.cancel when the client negotiated the consumer_cancel_notify capability. Modern clients enable it by default; very old clients may silently lose the subscription instead.

Does the channel close when this happens? No. Only the consumer subscription ends. You can re-declare the queue and call basic.consume again on the same channel.

How do I stop queues from being auto-deleted? Remove auto-delete and x-expires from the queue arguments, and avoid exclusive queues for shared long-lived consumers.

Will switching to quorum queues prevent cancels on node failure? Quorum queues survive node loss via leader election, so a single node failure does not make the queue unavailable. A leader change may still briefly cancel and require re-consume, but the queue persists.