☁️ OpenStack Error: Neutron L3 HA Failover Not Switching Router Master (keepalived)

Exact Error Message

$ openstack network agent list --router router-prod --long
+--------------+------------+-------------+-------+----------------+
| ID           | Agent Type | Host        | Alive | HA State       |
+--------------+------------+-------------+-------+----------------+
| 3f1a...      | L3 agent   | network-01  | :-)   | standby        |
| 7c9b...      | L3 agent   | network-02  | XXX   | active         |
+--------------+------------+-------------+-------+----------------+

The l3-agent / keepalived logs show VRRP never transitioning the healthy node to master:

INFO neutron.agent.l3.ha [-] Router 8d2c... transitioned to backup
Keepalived_vrrp[41822]: VRRP_Instance(VR_1) Received advert with lower
priority 50, ours 50, forcing new election
Keepalived_vrrp[41822]: (VR_1) Entering BACKUP STATE
Keepalived_vrrp[41822]: (VR_1) Now in FAULT state

What It Means

In Neutron L3 HA, each router is scheduled to two or more L3 agents. The agents run keepalived, which uses VRRP advertisements over a dedicated HA network to elect exactly one master (active) that owns the router’s gateway IP and floating IPs; the rest sit in standby. When the master fails, VRRP should promote a standby within seconds.

Failover “not working” means the promotion never happens: either both nodes think they are backup (no master, traffic blackholes), both claim master (split-brain, duplicate IPs and flapping), or the dead agent still shows HA State active while its Alive flag is XXX. The root cause is almost always the VRRP path — the HA network dropping multicast/VRRP packets — or keepalived stuck in FAULT state.

Common Causes

• The HA network blocks VRRP: multicast (224.0.0.18) or IP protocol 112 is filtered by security groups, an ML2 firewall, or the physical fabric.
• keepalived is in FAULT state because its tracked interface or track script is failing on the healthy node.
• Split-brain: the two agents cannot hear each other’s adverts, so both become master.
• The dead L3 agent’s HA State is stale in the database and never re-scheduled.
• MTU mismatch on the HA network fragments/drops VRRP adverts.
• Only one HA agent is actually alive, so there is no standby to promote.

Diagnostic Commands

Check agent liveness and current HA roles for the router:

openstack network agent list --router router-prod --long
openstack network agent list --agent-type l3

Inspect keepalived state inside the router namespace on each network node:

ip netns exec qrouter-<router-id> ip -o addr show | grep -w scope
ssh network-01 "grep -i 'state\|fault\|master\|backup' \
  /var/log/neutron/neutron-keepalived-state-change.log | tail"

Confirm VRRP adverts are actually crossing the HA network:

ip netns exec qrouter-<router-id> tcpdump -ni ha-<id> vrrp -c 5

Verify only one node owns the gateway IP (two owners = split-brain):

for h in network-01 network-02; do
  ssh $h "ip netns exec qrouter-<router-id> ip addr | grep <vip>"
done

Step-by-Step Resolution

1. Determine the failure mode from the agent list: no active (no master), two active (split-brain), or a dead node still marked active (stale schedule).

2. Fix the VRRP path first — it underlies most cases. Confirm the HA network passes multicast/protocol-112 and has a consistent MTU, then watch for adverts:

ip netns exec qrouter-<router-id> tcpdump -ni ha-<id> vrrp

If no adverts arrive on the standby, open VRRP on the HA network and any intervening firewall.

3. Clear a keepalived FAULT. It usually means the tracked interface is down; restart the L3 agent on the affected node so keepalived re-reads state:

systemctl restart neutron-l3-agent

4. For split-brain, restart keepalived on the node that should be standby so a single master wins the election, then re-verify only one owner of the VIP.

5. For a stale/dead agent that still shows active, remove the router from the dead agent and let Neutron reschedule it to a live one:

openstack network agent remove router --l3 <dead-agent-id> router-prod
openstack network agent add router --l3 <healthy-agent-id> router-prod

6. Confirm exactly one active master and that failover now works by draining the current master:

openstack network agent list --router router-prod --long

Prevention

• Ensure the HA (VRRP) network permits multicast and IP protocol 112 end-to-end, and keep its MTU consistent with the underlay.
• Schedule every HA router to at least two live agents and alert when a router has fewer than two alive.
• Monitor neutron-keepalived-state-change.log for FAULT transitions and duplicate-master events.
• Keep l3_ha = True and max_l3_agents_per_router sane (2–3); more agents mean more VRRP chatter, not more safety.
• Watch for MTU drift on the HA network after any fabric or MTU change, since fragmented adverts silently break elections.

Related Errors

• HA State: active on a XXX (dead) agent — a stale schedule that never re-elected.
• Duplicate IP / MAC flapping alarms on the gateway — the split-brain symptom.
• Router ... has no active HA agent — no master was elected at all.
• keepalived ... Now in FAULT state — a tracked-interface/track-script failure blocking promotion.

Frequently Asked Questions

Why doesn’t a standby take over when the master node dies? Almost always because VRRP adverts are not crossing the HA network — multicast or IP protocol 112 is blocked, or MTU drift is dropping them — so the standby never learns the master is gone and never promotes itself.

How do I tell if I have split-brain? Check the gateway/floating IP on each network node inside the qrouter- namespace. If two nodes both own the VIP, both keepalived instances elected themselves master; restart keepalived on the one that should be standby.

What does keepalived FAULT state mean? keepalived is tracking an interface or script that is failing, so it refuses to become master. Fix the underlying interface and restart neutron-l3-agent on that node to clear it.

Can I force a router onto a healthy agent? Yes — use openstack network agent remove router on the dead agent and openstack network agent add router on a live one to reschedule it, which triggers a fresh HA election.

Where can I find more Neutron fixes? Browse the full OpenStack guides for L3 HA, DHCP, and OVS troubleshooting, and grab a ready-made investigation prompt from the prompt library.