☁️ OpenStack Error: Open vSwitch agent 'XXX' dead — fix neutron-openvswitch-agent / ovs-vswitchd

Exact Error Message

$ openstack network agent list
+--------------+--------------------+-----------+-------+-------+----------------------------+
| ID           | Agent Type         | Host      | Alive | State | Binary                     |
+--------------+--------------------+-----------+-------+-------+----------------------------+
| 7c2f...      | Open vSwitch agent | compute-03| XXX   | UP    | neutron-openvswitch-agent  |
+--------------+--------------------+-----------+-------+-------+----------------------------+

# /var/log/neutron/neutron-openvswitch-agent.log
ERROR neutron.agent.common.ovs_lib [-] Unable to execute
    ['ovs-vsctl', '--timeout=10', ...]. Exception: ovs-vsctl: unix:/var/run/openvswitch/db.sock:
    database connection failed (Connection refused)
ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-]
    Error while processing VIF ports; MessagingTimeout

The Alive column showing XXX (a “dead” smiley) means Neutron server has not received a heartbeat from the agent within agent_down_time.

What It Means

The Neutron Open vSwitch agent (neutron-openvswitch-agent) programs the OVS bridges (br-int, br-tun, br-ex) that carry tenant and external traffic on each node. It depends on two OVS daemons: ovsdb-server (the configuration database, reached over db.sock) and ovs-vswitchd (the fast-path datapath). The agent also heartbeats to neutron-server over RabbitMQ.

An agent shows dead for one of two reasons: the agent process is stuck or crashed (so it stops heartbeating), or OVS itself is unresponsive (ovs-vswitchd/ovsdb-server down) so every ovs-vsctl call the agent makes fails. Either way, ports on that host stop being wired up: new instances get no connectivity and existing flows may break.

Common Causes

• ovsdb-server or ovs-vswitchd crashed or is hung; db.sock refuses connections.
• The agent lost its RabbitMQ connection, so heartbeats never reach neutron-server (MessagingTimeout).
• ovs-vswitchd is pegged/deadlocked after a kernel or OVS package upgrade without a datapath reload.
• Integration bridge br-int is missing or was deleted, so port setup fails.
• Clock skew makes the last heartbeat look older than agent_down_time.
• The agent is alive but wedged rebuilding flows on a very large number of ports.

Diagnostic Commands

Confirm whether OVS itself is healthy on the affected host:

sudo systemctl status ovs-vswitchd ovsdb-server
sudo ovs-vsctl show
sudo ovs-vsctl --version

Check the agent process and its recent log:

sudo systemctl status neutron-openvswitch-agent
sudo tail -f /var/log/neutron/neutron-openvswitch-agent.log

Look at the bridges and datapath the agent expects:

sudo ovs-vsctl list-br
sudo ovs-ofctl show br-int

Confirm what Neutron server thinks of the agent and its heartbeat timing:

openstack network agent show 7c2f... -c alive -c last_heartbeat_at -c configurations

Step-by-Step Resolution

1. Fix OVS first if ovs-vsctl show fails. Restart the OVS daemons before touching the Neutron agent (order matters — the agent needs a working db.sock):

sudo systemctl restart ovsdb-server ovs-vswitchd
sudo ovs-vsctl show   # must return the bridge list, not "Connection refused"

2. Verify the integration bridge exists; recreate it only if it is genuinely missing:

sudo ovs-vsctl list-br | grep br-int || sudo ovs-vsctl add-br br-int

3. Restart the Neutron OVS agent so it re-syncs flows against the now-healthy OVS:

sudo systemctl restart neutron-openvswitch-agent

4. Watch the agent complete its initial sync without db.sock or RPC errors:

sudo grep -E 'Agent .* out of sync|Configuration for devices up|MessagingTimeout' \
  /var/log/neutron/neutron-openvswitch-agent.log | tail

INFO neutron...ovs_neutron_agent [-] Agent rpc_loop - iteration completed. Processed ports.

5. If the agent is up but still marked dead, the problem is messaging. Check RabbitMQ reachability and clocks:

sudo rabbitmqctl list_connections | grep compute-03
sudo chronyc tracking

6. Confirm the agent is alive again:

openstack network agent list --host compute-03 --agent-type open-vswitch -c Alive -c State

| Alive | State |
| :-)   | UP    |

Prevention

• Alert on openstack network agent list showing any agent Alive = XXX for longer than agent_down_time.
• Reload the OVS datapath (force-reload-kmod) as part of any OVS package upgrade so ovs-vswitchd does not hang.
• Keep RabbitMQ healthy and highly available; most “dead” agents are actually a messaging outage.
• Run NTP/chrony everywhere so heartbeat timestamps are trustworthy.
• Do not manually delete br-int/br-tun; let the agent manage its bridges.

Related Errors

• neutron-server AgentNotFoundByTypeHost — the agent never registered on that host.
• MessagingTimeout in agent logs — a RabbitMQ/oslo.messaging outage, not an OVS crash.
• Port ... failed to bind — a binding failure downstream of a dead agent.
• ovs-vsctl: database connection failed — ovsdb-server is down, the root cause here.

Frequently Asked Questions

Why does the agent show dead even though the process is running? Because “dead” means Neutron server stopped receiving heartbeats. If the process is up, the cause is almost always a lost RabbitMQ connection or clock skew, not OVS.

In what order should I restart things? Restart ovsdb-server and ovs-vswitchd first, confirm ovs-vsctl show works, then restart neutron-openvswitch-agent. The agent cannot recover while OVS is unresponsive.

Will restarting the OVS agent drop tenant traffic? Existing datapath flows generally keep forwarding during a brief agent restart because ovs-vswitchd retains them, but avoid restarting the OVS daemons themselves during peak unless OVS is the fault.

How do I catch this before users do? Poll agent liveness and OVS health on a schedule; the automation patterns in the prompt library make that a one-liner. For more networking fixes, see the OpenStack guides.