☁️ OpenStack Error: Neutron DHCP Agent Down — Instances Not Getting IP Addresses

Exact Error Message

$ openstack network agent list --agent-type dhcp
+--------------------------------------+------------+-----------+-------------------+-------+-------+
| ID                                   | Agent Type | Host      | Availability Zone | Alive | State |
+--------------------------------------+------------+-----------+-------------------+-------+-------+
| a1b2c3d4-...                         | DHCP agent | network-1 | nova              | XXX   | UP    |
+--------------------------------------+------------+-----------+-------------------+-------+-------+

Inside the guest console, the boot log shows DHCP timing out:

[   34.812] cloud-init: Failed to bring up eth0.
[   34.815] dhcp: no lease, failing
[   41.002] eth0: DHCPDISCOVER sent, no DHCPOFFER received

And neutron-dhcp-agent.log stops sending heartbeats or logs dnsmasq failures:

neutron.agent.dhcp.agent [req-...] Unable to sync network state.
oslo.messaging MessagingTimeout: Timed out waiting for a reply to message ID ...

What It Means

The Alive = XXX marker means the Neutron server has not received a heartbeat from the DHCP agent within agent_down_time, so it considers the agent dead. When a DHCP agent is down, the dnsmasq processes it manages stop answering DHCPDISCOVER requests, and any instance that boots (or renews its lease) on a network served only by that agent fails to obtain an IP address. Existing instances keep their current lease until it expires, which is why the problem sometimes appears gradually.

The State = UP but Alive = XXX combination specifically means the agent is administratively enabled but not heartbeating — a process, messaging, or host problem rather than an intentional disable.

Common Causes

• The neutron-dhcp-agent service crashed or was stopped on the network node.
• RabbitMQ / messaging is unreachable, so the agent cannot heartbeat (MessagingTimeout).
• The dnsmasq processes for the network died or were never spawned in the agent’s namespace.
• The network node lost time sync, so heartbeats appear stale versus agent_down_time.
• Only one DHCP agent serves the network (dhcp_agents_per_network=1) and it failed, with no standby.

Diagnostic Commands

Confirm the agent’s alive state and which host it runs on:

openstack network agent list --agent-type dhcp --long

On the network node, check the service and its logs:

sudo systemctl status neutron-dhcp-agent
sudo tail -n 40 /var/log/neutron/neutron-dhcp-agent.log

Check whether dnsmasq is actually running for the network’s namespace:

ip netns list | grep qdhcp
sudo ip netns exec qdhcp-<network-id> ps aux | grep dnsmasq

Verify messaging reachability from the node (RabbitMQ heartbeats):

sudo grep -i -E 'MessagingTimeout|AMQP|rabbit' /var/log/neutron/neutron-dhcp-agent.log | tail

See which networks the (dead) agent is hosting:

openstack network agent show <agent-id> -c alive -c host
openstack network list --agent <agent-id> 2>/dev/null || \
  openstack network agent list --network <network-id>

Step-by-Step Resolution

1. Confirm the agent is down and identify its host:

openstack network agent list --agent-type dhcp

2. On that host, restart the agent and confirm it comes up:

sudo systemctl restart neutron-dhcp-agent
sudo systemctl status neutron-dhcp-agent

3. If messaging was the cause, verify RabbitMQ is healthy and reachable, then let the agent re-register:

sudo rabbitmqctl node_health_check

4. If dnsmasq did not respawn for a network, force the agent to re-sync by removing and re-adding the network to the agent:

openstack network agent remove network --dhcp <agent-id> <network-id>
openstack network agent add network --dhcp <agent-id> <network-id>

5. If the whole host is unrecoverable, move its networks to a healthy DHCP agent so instances get IPs immediately:

openstack network agent add network --dhcp <healthy-agent-id> <network-id>

6. Confirm the agent is alive again and dnsmasq is answering:

openstack network agent list --agent-type dhcp
sudo ip netns exec qdhcp-<network-id> ps aux | grep dnsmasq

7. Reboot or renew a stuck instance and verify it now gets a lease:

openstack server reboot --hard test-vm
openstack server show test-vm -c addresses -f value

Prevention

• Run at least two DHCP agents per network (dhcp_agents_per_network=2) so a single agent failure does not black-hole DHCP.
• Keep NTP/chrony healthy on network nodes; clock skew makes live agents look dead against agent_down_time.
• Monitor openstack network agent list and alert on any Alive=XXX, plus monitor RabbitMQ from the network nodes.
• Watch for orphaned qdhcp-* namespaces and missing dnsmasq processes as an early warning.
• Use L3/DHCP HA and automatic rescheduling (allow_automatic_dhcp_failover=true) so networks re-home off a dead agent.

Related Errors

• agent ... is not alive for L3 or OVS agents — the same heartbeat failure on other Neutron agents.
• No more IP addresses available on network — an exhausted subnet pool, not a dead agent.
• MessagingTimeout — the RabbitMQ/AMQP cause behind many agent-down conditions.
• cloud-init: no lease, failing — the guest-side symptom of the DHCP outage.

Frequently Asked Questions

Why do existing instances keep working while new ones fail? Running instances hold their DHCP lease until it expires, so they stay reachable, but any instance that boots or renews while the agent is down cannot get an offer and fails with no lease.

What does Alive = XXX actually mean? The Neutron server has not received a heartbeat within agent_down_time. It usually means the agent process died or messaging is broken — check the service and RabbitMQ. For scripting these health checks, the OpenStack prompts library has agent-monitoring workflows.

How do I move a network off a dead DHCP agent? Use openstack network agent add network --dhcp <healthy-agent-id> <network-id> to schedule it onto a healthy agent, which spawns a fresh dnsmasq and restores DHCP.

Should I run more than one DHCP agent per network? Yes. Set dhcp_agents_per_network=2 so a single agent failure leaves a running standby, avoiding exactly this outage.

Why did the agent look dead even though the process was running? Clock skew on the network node can make heartbeats appear stale relative to agent_down_time. Fix NTP/chrony and the agent will report alive again. For more networking fixes, see the OpenStack guides.