🚨 RabbitMQ Error Guide: 'federation link ... unavailable' Upstream Connection Failure

Exact Error Message

A federation link cannot reach or stay connected to its upstream, and the broker log reports the link cycling between starting and failing:

2026-06-29 13:05:44.118 [warning] <0.2241.0> Federation link for upstream 'dc2-upstream'
on exchange 'orders' reported status: unavailable

2026-06-29 13:05:44.121 [error] <0.2241.0> Federation upstream 'dc2-upstream' connection failed:
{error,{auth_failure,"Refused"}}

2026-06-29 13:06:14.890 [error] <0.2255.0> CRASH REPORT Process <0.2255.0> with 0 neighbours
exited with reason: {shutdown,{server_initiated_close,403,
  <<"ACCESS_REFUSED - access to exchange 'orders' in vhost '/' refused for user 'fed'">>}}

Network-level failures look slightly different:

2026-06-29 13:11:02.337 [error] <0.2310.0> Federation upstream 'dc2-upstream' connection failed:
{error,{econnrefused,"10.20.0.7:5672"}}

What the Error Means

Federation links one broker’s exchange or queue to a matching object on a remote “upstream” broker, lazily copying messages across a WAN. Each link opens an ordinary AMQP connection from the downstream broker to the upstream URI defined in an upstream parameter and selected by a policy. A status of unavailable means that connection is not currently established: the link tried to connect and failed, or it connected and was then closed by the upstream.

The reason wrapped in the error tells you which layer failed — auth_failure/403 ACCESS_REFUSED is credentials or permissions, econnrefused/etimedout/nxdomain is network or DNS, and a server_initiated_close with a resource error is a missing exchange/queue or vhost on the upstream. The link retries on a backoff, so you see repeated unavailable transitions rather than a single fatal error.

Common Causes

• Wrong upstream credentials. The user/password embedded in the upstream URI is wrong or rotated, giving auth_failure.
• Missing permissions on the upstream. The federation user lacks read/configure on the upstream exchange/queue or vhost (403 ACCESS_REFUSED).
• Network or DNS failure. Firewall, routing, or name resolution blocks the connection (econnrefused, etimedout, nxdomain).
• Upstream object does not exist. The matching exchange or queue (or vhost) is absent on the upstream.
• No policy selects the link. An upstream parameter exists but no policy with federation-upstream / federation-upstream-set matches the local object, so no link runs.
• TLS mismatch. amqps:// URI with an untrusted or missing certificate chain.

How to Reproduce the Error

Define an upstream with bad credentials and a policy that selects it:

# upstream URI with a wrong password
rabbitmqctl set_parameter federation-upstream dc2-upstream \
  '{"uri":"amqp://fed:WRONGPASS@10.20.0.7:5672","expires":3600000}'

# policy applies the upstream to exchanges named orders.*
rabbitmqctl set_policy --apply-to exchanges fed-orders "^orders" \
  '{"federation-upstream":"dc2-upstream"}'

# the link now appears as 'unavailable' with an auth_failure in the log

Diagnostic Commands

# Status of every federation link: running vs starting vs error
rabbitmqctl eval 'rabbit_federation_status:status().'

[[{exchange,<<"orders">>},{upstream,<<"dc2-upstream">>},
  {status,error},{error,{auth_failure,"Refused"}}]]

# Inspect the configured upstream (URI, expires, ack-mode)
rabbitmqctl list_parameters | grep federation-upstream

# Confirm a policy actually selects the link
rabbitmqctl list_policies

/  fed-orders  exchanges  ^orders  {"federation-upstream":"dc2-upstream"}  0

# Pull federation errors from the log
sudo grep -iE 'Federation (link|upstream)|ACCESS_REFUSED' \
  /var/log/rabbitmq/rabbit@$(hostname -s).log | tail -15

# Test raw connectivity from the downstream host to the upstream
nc -vz 10.20.0.7 5672
getent hosts upstream.dc2.internal

# Validate the upstream credentials against the upstream broker's API
curl -s -u fed:secret http://10.20.0.7:15672/api/whoami

If nc fails, it is network/DNS. If nc succeeds but /api/whoami 401s, the credentials are wrong. If both succeed, suspect permissions or a missing upstream object.

Step-by-Step Resolution

1. Read the link status reason. rabbit_federation_status:status() gives the exact error per link — auth_failure, econnrefused, a 403 ACCESS_REFUSED, or a missing-object close.

2. Fix credentials for auth_failure. Re-set the upstream parameter with the correct user/password in the URI:

   rabbitmqctl set_parameter federation-upstream dc2-upstream \
     '{"uri":"amqp://fed:CORRECTPASS@10.20.0.7:5672"}'

   The link picks up the new parameter and reconnects automatically.

3. Grant upstream permissions for 403 ACCESS_REFUSED. On the upstream broker, give the federation user read on the source object and its vhost (set_permissions -p <vhost> fed ...).

4. For econnrefused/etimedout/nxdomain, fix the network path: open the firewall to the upstream’s 5672/5671, correct DNS, and verify routing with nc -vz.

5. Create the missing upstream object. Ensure the matching exchange/queue and vhost exist on the upstream; federation does not create the source side for you.

6. Confirm a policy selects the link. If list_policies shows nothing matching your object, add a policy with federation-upstream or federation-upstream-set; an upstream parameter alone does nothing.

7. For TLS, use amqps:// and ensure the downstream trusts the upstream’s CA via the upstream’s ca_cert/cert/key in the URI query or the broker’s TLS config.

Verify recovery by re-running the status command until the link shows running.

Prevention and Best Practices

• Use a dedicated, least-privilege federation user on the upstream and rotate its credentials with automation that re-sets the upstream parameter in the same step.
• Define upstream sets for multi-upstream topologies so policies reference a stable set name rather than individual upstreams.
• Monitor rabbit_federation_status:status() and alert when any link leaves running, so a WAN blip or credential rotation surfaces immediately.
• Keep expires and message-ttl on upstreams sane so abandoned links clean up instead of accumulating.
• Pin firewall rules between data centres explicitly to the broker ports, and prefer DNS names with health-aware records over raw IPs.
• Always use TLS for cross-site federation and manage the trust chain alongside certificate renewal.

Related Errors

• shovel worker terminated — the other message-moving plugin; similar URI/credential failures but a different supervisor and status command.
• inet error etimedout / stale connection — the underlying TCP failure a federation link can wrap as etimedout.
• epmd nxdomain / host resolution — DNS failures that also break upstream URIs using hostnames.
• ACCESS_REFUSED - Login was refused — the AMQP auth failure the link reports as auth_failure.

See the wider RabbitMQ troubleshooting series.

Frequently Asked Questions

Why does my upstream parameter exist but no link runs? A policy must select the object with federation-upstream or federation-upstream-set. Without a matching policy, the upstream is inert.

Where do I grant the federation user permissions — upstream or downstream? On the upstream broker, because that is where the link reads the source exchange/queue. A 403 ACCESS_REFUSED always refers to the upstream.

Does federation create the source exchange or queue for me? No. The matching object and vhost must already exist on the upstream; only the downstream side is created by the policy.

How do I see exactly why a link is down? Run rabbitmqctl eval 'rabbit_federation_status:status().'; the error field gives the precise reason per link.

Will federation auto-recover after I fix credentials? Yes. Re-setting the upstream parameter triggers a reconnect on backoff; you do not need to restart the broker.