🐰 RabbitMQ Error: Federation Link Down / Upstream Not Running

Exact Error Message

A federation link that cannot establish or stay connected reports a non-running status. rabbitmqctl federation_status shows it, and the log records the reason:

Status of federation on node rabbit@mq-01 ...
[{upstream,<<"dr-upstream">>},
 {status,down},
 {local_connection,<<"amqp://localhost">>},
 {upstream_queue,<<"orders">>},
 {reason,{shutdown,{connection_refused,"amqp://mq-dr.internal:5672",econnrefused}}}]

2026-07-17 10:02:41.663 [warning] <0.877.0> Federation upstream 'dr-upstream'
    for exchange 'events' is down: connection refused, retrying in 5000 ms

You may also see status,down with reason,{error,not_found} when the upstream itself was never declared, or an empty status list when no policy matches any object.

What It Means

Federation makes a downstream broker transparently pull messages from an upstream exchange or queue on another broker. A federation link is the live AMQP connection that does that pulling. status,down means the link exists in intent — an upstream is declared and a policy selects an object to federate — but the link is not currently running.

The most common cause is that the upstream connection cannot be established (refused, unauthorized, or unresolved). A second, subtler cause is that the plumbing is incomplete: the rabbitmq_federation plugin is enabled but no federation-upstream parameter exists, or no policy matches the exchange/queue you expected, so no link is even attempted.

Common Causes

• The upstream broker is unreachable — port 5672/5671 blocked, host down, or wrong URI (connection_refused).
• The federation-upstream parameter is missing or misnamed, so the policy references an upstream that doesn’t exist (not_found).
• No policy matches the exchange or queue, so a link is never created and federation_status is empty.
• Wrong credentials in the upstream URI (access_refused in the reason term).
• The rabbitmq_federation plugin is not enabled on the downstream node.
• A vhost mismatch — the upstream URI omits or wrongly encodes the vhost (%2F).

Diagnostic Commands

Check link status on the downstream node:

rabbitmqctl federation_status

Confirm the plugin is actually enabled:

rabbitmq-plugins list -e | grep federation

List the upstream definitions and the policies that should select objects to federate:

# Upstreams (component federation-upstream) and sets
rabbitmqctl list_parameters | grep federation

# Policies — check one actually matches your exchange/queue
rabbitmqctl list_policies

Test reachability from the downstream broker to the upstream, exactly as the link would:

nc -vz mq-dr.internal 5672
getent hosts mq-dr.internal

Follow the log to capture the live retry reason:

sudo journalctl -u rabbitmq-server -f | grep -i federation

Step-by-Step Resolution

1. Read the reason term from federation_status. connection_refused is a reachability problem (go to step 2); not_found means the upstream is missing (step 3); an empty status means no policy matches (step 4).

2. For a refused connection, prove the port from the downstream broker and open it if blocked:

nc -vz mq-dr.internal 5672
rabbitmq-diagnostics -n rabbit@mq-dr listeners   # on the upstream, if reachable

Fix the firewall/security group or correct the host in the upstream URI.

3. For not_found, (re)declare the upstream the policy points at:

rabbitmqctl set_parameter federation-upstream dr-upstream \
  '{"uri":"amqp://mq-dr.internal:5672/%2F","expires":3600000}'

4. For an empty status, make sure a policy selects your object and references the upstream by name:

rabbitmqctl set_policy federate-events '^events$' \
  '{"federation-upstream":"dr-upstream"}' \
  --priority 1 --apply-to exchanges

5. If the reason is access_refused, fix the credentials embedded in the upstream URI (user, password, and vhost), then reset the parameter as in step 3.

6. If the plugin is not listed as enabled, enable it and restart is not required for federation:

rabbitmq-plugins enable rabbitmq_federation rabbitmq_federation_management

7. Confirm the link recovers:

rabbitmqctl federation_status

The upstream should now report status,running with no retry lines in the log.

Prevention

• Alert on any federation_status entry that is not running; a down link moves no messages and is otherwise silent.
• Keep upstream parameters and the matching policies in configuration management so names stay in sync — a policy that names a non-existent upstream is a common footgun.
• Verify upstream reachability (nc -vz) from the downstream broker before declaring a link.
• Set an expires or message-ttl on federated upstream queues so a long outage doesn’t accumulate unbounded backlog.
• Document which vhost each upstream targets and encode it correctly (%2F for the default vhost).

Related Errors

• Shovel connection refused — the same reachability failure for a shovel instead of a federation link.
• access_refused — the upstream connection is reached but credentials are rejected.
• no upstream set — a policy uses federation-upstream-set that isn’t defined.
• nxdomain — the upstream URI names a host that does not resolve.

Frequently Asked Questions

Why is federation_status empty even though I enabled the plugin? Enabling the plugin and declaring an upstream is not enough — a policy must match an exchange or queue and reference the upstream. With no matching policy, no link is created and the status list is empty.

What’s the difference between a down link and a missing one? A down link means an upstream and policy exist but the connection can’t be established (usually reachability or auth). A missing link means no policy selects any object, so nothing is attempted at all.

Do I fix the upstream or the downstream broker? Run federation_status and the diagnostics on the downstream (the broker pulling messages). The URI, policy, and plugin all live there; the upstream just needs to be reachable and accept the connection.

How do I point a policy at the right upstream? Set the policy’s federation-upstream (single) or federation-upstream-set (group) to the exact parameter name you declared with set_parameter federation-upstream. A name mismatch produces a not_found reason.

Should I use federation or a shovel? Federation transparently links exchanges/queues across brokers and follows bindings; a shovel moves messages between two specific queues. For more messaging patterns, see the RabbitMQ guides or grab a ready-made prompt from the RabbitMQ prompt library.