🐰 RabbitMQ Error: Shovel 'connection_refused' Cannot Reach Source or Destination

Exact Error Message

A shovel that cannot reach one side of its link never reaches the running state. rabbitmqctl shovel_status reports it as terminated or starting, and the broker log shows the connection failure:

Shovel 'orders-to-dr' (dynamic): failed to connect to the destination:
    {shutdown,{connection_refused,"amqp://mq-dr.internal:5672",econnrefused}}

2026-07-17 09:14:02.118 [warning] <0.912.0> Shovel 'orders-to-dr' will restart in 5 seconds
2026-07-17 09:14:07.221 [error] <0.940.0> Shovel 'orders-to-dr' could not connect:
    econnrefused (connection refused by mq-dr.internal:5672)

The management UI shows the same shovel with a red terminated status and a reason of econnrefused. Static shovels defined in rabbitmq.conf fail identically, only without a management entry.

What It Means

A shovel is a background connection that consumes from a source queue/exchange on one broker and republishes to a destination on another. It is a plain AMQP client running inside the broker. connection_refused / econnrefused means the TCP connect to the URI it was given was actively rejected — nothing is listening on that host and port, or a firewall dropped the SYN with a reset.

This is a reachability problem, not a messaging problem. The shovel keeps retrying on its reconnect interval, so you see the same line repeat every few seconds. Until it connects, no messages move, and on the source broker the shovel’s queue quietly grows.

Common Causes

• The destination (or source) broker’s AMQP listener is down or bound to 127.0.0.1 only.
• A firewall or security group blocks port 5672 (or 5671 for TLS) between the two brokers.
• The URI names the wrong host, port, or vhost — a typo, an internal DNS name that doesn’t resolve from the shovel’s broker, or a missing %2F-encoded vhost.
• TLS mismatch: the URI uses amqp:// (5672) against a TLS-only listener, or amqps:// without the right CA/cert config.
• Wrong credentials look similar but surface as access_refused, not connection_refused — check which you actually have.

Diagnostic Commands

Confirm the shovel’s current state and the exact reason from a node that hosts it:

rabbitmqctl shovel_status

If your build predates that subcommand, read the status term directly:

rabbitmqctl eval 'rabbit_shovel_status:status().'

Show the stored definition so you can see the real URIs (dynamic shovels are stored as runtime parameters):

rabbitmqctl list_parameters | grep shovel

Test raw reachability from the broker that runs the shovel to the far side:

# TCP reachability of the AMQP port
nc -vz mq-dr.internal 5672

# DNS resolves from this host?
getent hosts mq-dr.internal

Watch the log while it retries to capture the precise failure term:

sudo journalctl -u rabbitmq-server -f | grep -i shovel

Step-by-Step Resolution

1. Identify which side is refused. Read the log line: failed to connect to the destination vs the source tells you which URI to fix. Note the host and port in the connection_refused tuple.

2. Prove reachability with nc -vz <host> <port> from the broker running the shovel — not from your laptop. A shovel runs inside the broker process, so its network path is the broker’s, which may differ from yours.

3. If the port is refused, fix the listener or the firewall. Confirm the far broker is actually listening on that interface:

# On the destination broker
rabbitmq-diagnostics listeners
sudo ss -ltnp | grep -E ':5672|:5671'

Open 5672/5671 in the security group both directions if it is a network block.

4. If the host doesn’t resolve, fix DNS or use an address the broker can reach. Then correct the URI. For a dynamic shovel, redeclare it (this replaces the parameter atomically):

rabbitmqctl set_parameter shovel orders-to-dr \
  '{"src-uri":"amqp://mq-01.internal:5672/%2F",
    "src-queue":"orders",
    "dest-uri":"amqp://mq-dr.internal:5672/%2F",
    "dest-queue":"orders"}'

5. If the far side is TLS-only, switch the scheme to amqps:// and supply the TLS options in the URI query string (cacertfile, certfile, keyfile, verify=verify_peer), then test port 5671 with nc -vz.

6. Remove and recreate a wedged dynamic shovel if it will not pick up the new definition:

rabbitmqctl clear_parameter shovel orders-to-dr
# then set_parameter again as above

7. Confirm it reaches running:

rabbitmqctl shovel_status

The shovel should show running with a populated src_uri/dest_uri and no reconnect lines in the log.

Prevention

• Health-check both broker endpoints (a simple nc -vz or AMQP connect) from monitoring before declaring a cross-site shovel.
• Alert on rabbitmqctl shovel_status reporting anything other than running; a terminated shovel is silent otherwise.
• Store shovel URIs in configuration management so a typo is reviewed, not hand-typed.
• Prefer stable DNS names or service endpoints over IPs, and verify they resolve from the broker host, not just the operator’s shell.
• Keep source and destination on matching TLS settings; document which port (5672 vs 5671) each listener uses.

For reusable prompts that turn a shovel status dump into a concrete fix, see the RabbitMQ prompt library.

Related Errors

• access_refused — the TCP connect succeeds but the shovel’s credentials are rejected; a login/permissions problem, not reachability.
• federation link down — the federation plugin’s equivalent failure mode for upstream links.
• unknown host / nxdomain — the shovel URI names a host that does not resolve at all.

Frequently Asked Questions

Why does the shovel keep retrying instead of just failing? Shovels are designed to be self-healing, so on any connection error they wait the reconnect interval and try again. That is why you see the same connection_refused line every few seconds until the endpoint is reachable.

Is connection_refused a credentials problem? No. connection_refused/econnrefused happens at TCP connect, before authentication. Wrong credentials produce access_refused after the connection is established.

Do I run the diagnostics on the source or destination broker? Run shovel_status and the reachability tests on the broker that hosts the shovel — usually the source. Its network path is what matters, not your workstation’s.

How do I change a dynamic shovel’s URI safely? Re-run rabbitmqctl set_parameter shovel <name> ... with the corrected definition; it replaces the parameter atomically and restarts the shovel. Clear and recreate it only if it refuses to pick up the change.

Should I use a shovel or federation for cross-site links? Shovels move messages between specific queues and are simplest for point-to-point replication; federation links exchanges/queues more transparently. For more patterns and fixes, see the RabbitMQ guides.