🚨 RabbitMQ Error Guide: 'Shovel ... terminated' Worker Failure and Lost Upstream

Exact Error Message

A shovel fails to start or dies after starting, and the log records the worker termination with a reason:

2026-06-29 10:22:03.551 [error] <0.1904.0> Shovel 'orders-to-archive' failed to connect to
source amqp://app@10.0.6.4:5672: {auth_failure,"ACCESS_REFUSED - Login was refused
using authentication mechanism PLAIN."}

2026-06-29 10:22:33.770 [error] <0.1904.0> CRASH REPORT Process <0.1904.0> with 0 neighbours
exited with reason: {shutdown,{shovel,'orders-to-archive',
  {error,{server_initiated_close,404,
    <<"NOT_FOUND - no queue 'orders.archive' in vhost '/'">>}}}}

2026-06-29 10:23:04.012 [warning] <0.1880.0> Shovel 'orders-to-archive' terminated;
restarting in 5000 ms (reconnect-delay)

A lost destination mid-run looks like:

2026-06-29 10:40:18.229 [error] <0.2050.0> Shovel 'orders-to-archive' destination connection
lost: {error,closed}; worker terminated, will reconnect

What the Error Means

A shovel is a worker that consumes from a source broker/queue and republishes to a destination broker/exchange, reliably moving messages between brokers or vhosts. Each shovel runs as a supervised worker that opens two AMQP connections (source and destination) from the shovel host. When either connection cannot be established or is lost, the worker terminates and the supervisor restarts it after reconnect-delay. That is why you see “terminated; restarting” on a loop rather than a clean stop.

The reason in the crash report localises the fault: auth_failure/ACCESS_REFUSED is credentials, 404 NOT_FOUND is a missing queue/exchange, econnrefused/closed is the network or a dropped peer, and a config error means the shovel definition itself is invalid. Dynamic shovels (declared via parameters/policy at runtime) and static shovels (in rabbitmq.conf) fail the same way but are inspected and fixed differently.

Common Causes

• Bad source or destination credentials. The user/password in a src-uri/dest-uri is wrong or rotated (auth_failure).
• Missing queue or exchange. The source queue or destination exchange/queue does not exist (404 NOT_FOUND).
• Network failure to a remote endpoint. Firewall/routing/DNS blocks the source or destination URI (econnrefused, etimedout, closed).
• Insufficient permissions. The shovel user lacks read on the source or write/configure on the destination.
• Invalid shovel definition. Malformed JSON, an unknown field, or add-forward-headers/ack-mode set incorrectly.
• Destination overload or alarm. The destination broker has a memory/disk alarm, closing the publishing connection.
• Static vs dynamic confusion. Editing rabbitmq.conf for a shovel that was actually declared dynamically (or vice versa), so changes never apply.

How to Reproduce the Error

Declare a dynamic shovel whose destination queue does not exist:

rabbitmqctl set_parameter shovel orders-to-archive \
  '{"src-uri":"amqp://","src-queue":"orders",
    "dest-uri":"amqp://","dest-queue":"orders.archive",
    "ack-mode":"on-confirm"}'

# orders.archive was never declared, so the worker terminates with 404 NOT_FOUND

Or point a shovel at an unreachable destination URI to get a network termination:

rabbitmqctl set_parameter shovel bad-dest \
  '{"src-uri":"amqp://","src-queue":"orders",
    "dest-uri":"amqp://guest@10.99.99.99:5672","dest-queue":"x"}'
# worker loops on econnrefused / terminated; restarting

Diagnostic Commands

# Status of every shovel: running, starting, or terminated with a reason
rabbitmqctl shovel_status

name              type     state      reason
orders-to-archive dynamic  terminated {error,{server_initiated_close,404,...}}

# Inspect the dynamic shovel definition (URIs, queues, ack-mode)
rabbitmqctl list_parameters | grep shovel

# Confirm the shovel plugin is enabled
rabbitmq-plugins list -e | grep -i shovel

# Pull shovel errors from the log
sudo grep -iE 'Shovel|server_initiated_close|auth_failure' \
  /var/log/rabbitmq/rabbit@$(hostname -s).log | tail -15

# Verify the destination queue/exchange actually exists
rabbitmqctl list_queues name | grep orders.archive

# Test reachability and credentials of a remote endpoint
nc -vz 10.0.6.4 5672
curl -s -u app:secret http://10.0.6.4:15672/api/whoami

rabbitmqctl shovel_status is the single most useful command — its reason column gives the exact termination cause for each shovel.

Step-by-Step Resolution

1. Read shovel_status. Identify the failing shovel, whether it is dynamic or static, and the termination reason.

2. For auth_failure, correct the credentials in the URI and re-declare the dynamic shovel:

   rabbitmqctl set_parameter shovel orders-to-archive \
     '{"src-uri":"amqp://app:CORRECT@10.0.6.4:5672","src-queue":"orders",
       "dest-uri":"amqp://","dest-queue":"orders.archive"}'

   Re-setting the parameter restarts the worker with the new config.

3. For 404 NOT_FOUND, create the missing object. Declare the destination queue/exchange (or source queue) on the correct broker and vhost; shovels do not auto-create endpoints unless the definition says so.

4. For network reasons (econnrefused/etimedout/closed), verify the path with nc -vz, fix firewall/DNS to the remote broker, and confirm the destination broker is not under a resource alarm.

5. For permission errors, grant the shovel user read on the source and write/configure on the destination vhost.

6. For an invalid definition, fix the JSON (valid fields, correct ack-mode of on-confirm/on-publish/no-ack) and re-set the parameter.

7. For a static shovel, edit the shovel.shovels.* block in rabbitmq.conf and restart the node — runtime set_parameter will not touch a config-defined shovel, which is a common source of “my change did nothing.”

Verify with rabbitmqctl shovel_status until the shovel shows running.

Prevention and Best Practices

• Prefer dynamic shovels declared via parameters/policy so you can fix and restart them without a broker restart; reserve static shovels for fixed infrastructure.
• Use a dedicated, least-privilege shovel user and rotate its credentials with automation that re-declares the shovel parameter in the same step.
• Pre-create source and destination objects (or set explicit declare options) so a deploy never races ahead of its queues.
• Use ack-mode: on-confirm for at-least-once delivery so a terminated worker does not lose in-flight messages.
• Monitor rabbitmqctl shovel_status and alert on any non-running state; a looping shovel silently stalls a pipeline.
• Watch the destination broker for memory/disk alarms, which close the publishing connection and terminate the worker.

Related Errors

• federation upstream unavailable — the sibling message-moving plugin; similar URI/credential failures but a different status command and model.
• ACCESS_REFUSED - Login was refused — the AMQP auth failure a shovel reports as auth_failure.
• resource alarm set (memory/disk) — a destination alarm that closes the shovel’s publishing connection.
• inet error etimedout / stale connection — the TCP-level failure behind a network termination.

More walkthroughs are in the RabbitMQ guides.

Frequently Asked Questions

Why does my shovel keep terminating and restarting every few seconds? The worker hits a fatal condition (bad creds, missing queue, unreachable peer) and the supervisor restarts it after reconnect-delay. Fix the reason shown in shovel_status to stop the loop.

My config change had no effect — why? You likely edited rabbitmq.conf for a shovel that was declared dynamically (or set a parameter for a static one). Static shovels need a restart; dynamic shovels are changed with set_parameter.

Does a shovel create the destination queue automatically? Not by default. Pre-create the endpoint or set explicit declaration options in the definition.

Will I lose messages when the worker terminates? With ack-mode: on-confirm (or on-publish), unconfirmed messages are redelivered after reconnect. With no-ack you can lose in-flight messages.

How do I see the exact failure reason? Run rabbitmqctl shovel_status; the reason column carries the termination cause for each shovel.