🔩 Debugging Ironic Bare Metal Provisioning in OpenStack

Ironic provisions bare metal as if it were a VM — same Nova API, same flavors, but the “instance” is a physical server you image over the network. That’s enormously powerful and enormously more fragile, because instead of a hypervisor you’re now depending on PXE, DHCP, TFTP, IPMI, and the firmware quirks of whatever hardware you bought. After years of running Ironic, I can tell you the failures are almost always at the boring physical layer, and the Ironic logs will happily send you looking everywhere else. Here’s the discipline that keeps it sane.

The state machine is your map

Ironic drives every node through a state machine: enroll → manageable → available → deploying → active, with cleaning and error states woven through. Every problem is “node stuck in state X,” and the state tells you which subsystem to suspect. Always start here:

openstack baremetal node list \
  --fields uuid name provision_state power_state maintenance last_error
openstack baremetal node show <node> -f value -c last_error

The last_error field is the single most useful thing in Ironic. It carries the conductor’s last failure verbatim — a timeout, an IPMI auth failure, a deploy ramdisk that never phoned home. Read it before you theorize.

Step 1: Trust IPMI before anything else

Ironic’s power and boot-device control runs over IPMI (or Redfish on newer gear). If that’s flaky, everything downstream is flaky. Test it out-of-band, bypassing Ironic entirely:

ipmitool -I lanplus -H <bmc-ip> -U <user> -P <pass> power status
ipmitool -I lanplus -H <bmc-ip> -U <user> -P <pass> chassis bootdev pxe

If power status is slow or intermittent, that’s your bug — a node that won’t reliably power-cycle will randomly fail deploys and cleaning. Common causes: BMC needs a cold reset (ipmitool mc reset cold), wrong privilege level, or a shared NIC where the BMC and the OS fight over the same port. On Redfish, confirm the driver address and that the BMC’s TLS cert is trusted.

Step 2: When PXE won’t boot

The deploy and cleaning ramdisks reach the node over PXE, which means DHCP must offer the right next-server and bootfile, and TFTP/HTTP must serve them. Most “node stuck in deploying” cases are PXE failures. Verify the provisioning network plumbing:

# On the conductor / provisioning network:
journalctl -u ironic-conductor -f
tcpdump -i <prov-iface> -n port 67 or port 69

That tcpdump tells you immediately whether the node is even sending DHCP discovers and whether it’s getting an offer. No discover means the NIC didn’t PXE-boot (BIOS boot order, or bootdev pxe didn’t stick). Offer but no TFTP read means a firewall or wrong tftp_root. With UEFI hardware, the classic trap is serving a BIOS bootloader to a UEFI node — confirm ipxe vs pxelinux and the right bootfile_name for the firmware mode.

Step 3: Nodes stuck in cleaning

Automated cleaning runs after every deprovision — it wipes disks and optionally updates firmware via the cleaning ramdisk. A node wedged in cleaning or clean_wait usually means the cleaning ramdisk never booted (same PXE issues as above) or a disk-wipe is genuinely slow on a huge spinning disk. Check:

openstack baremetal node show <node> -f value -c provision_state
openstack baremetal node show <node> -f value -c last_error

If it’s truly stuck and you’ve confirmed nothing is progressing, move it out of the way carefully:

openstack baremetal node maintenance set <node> \
  --reason "stuck in clean_wait, investigating"
openstack baremetal node abort <node>

abort cancels the clean step; maintenance set keeps Nova from scheduling onto it while you work. Never just delete the node object — that orphans ports and leaves the physical box in an unknown state.

Step 4: Match the deploy interface to your hardware

Ironic supports direct deploy (the agent pulls the image over HTTP and writes it locally — fast, the modern default) and the older iscsi flow. If deploys are slow or failing at the image-write step, confirm the deploy interface and that the agent ramdisk can reach the image store:

openstack baremetal node show <node> -f value -c deploy_interface

For the direct interface, the node needs network reach to the Swift/HTTP image location and enough RAM to stage the agent. A node that boots the agent ramdisk but fails to write the image is almost always a connectivity or disk-detection problem, not an Ironic bug — last_error will say which.

Step 5: Validate the whole node up front

Before you ever try to deploy, validate:

openstack baremetal node validate <node>

This checks power, management, boot, and deploy interfaces and tells you which are unconfigured. A node that fails validate will never deploy cleanly — fix the failing interface (usually power credentials or a missing deploy_kernel/deploy_ramdisk) first.

Where AI helps

Ironic failures span Ironic, Neutron (the provisioning network), DHCP, and physical firmware, and the conductor log is noisy. I’ll paste the node’s last_error, the conductor log tail, and the tcpdump summary into a model and ask it to localize the failure to one layer — IPMI, PXE/DHCP, ramdisk, or image write — and propose the read-only confirmation for each. It’s reliably good at recognizing a UEFI/BIOS bootloader mismatch from the TFTP request pattern.

Keep a saved Ironic triage prompt for the 3am deploy that won’t take, and browse our other OpenStack guides. The model reads the errors; you run every ipmitool and baremetal command yourself after you’ve understood it — bare metal is unforgiving of blind commands.

Generated commands are assistive, not authoritative. Always verify against your own hardware and deployment before running anything in production.