☁️ OpenStack Error: Cinder Volume Detach Fails 'device is busy' / Stuck in-use

Exact Error Message

$ openstack server remove volume my-instance 4d9c1b2a-...
Failed to detach volume 4d9c1b2a-... from server my-instance:
Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400)

Inside the guest and in the nova-compute log, the underlying cause is visible:

libvirt.libvirtError: internal error: unable to execute QEMU command 'device_del':
    Device vdb is in use

umount: /mnt/data: target is busy.

The volume then often lands in a detaching state and never returns to available:

$ openstack volume show 4d9c1b2a-... -c status -c attachments
| status      | detaching |

What It Means

A Cinder volume can only detach cleanly when nothing inside the guest is still using the block device. If a filesystem is mounted, a process has the device open, or the volume is part of an active LVM/RAID group, the guest kernel refuses to release it and QEMU reports Device is in use. Nova then cannot complete the detach, and the volume can get stuck in detaching (or stay in-use with a stale attachment).

This is almost always a guest-side problem, not a Cinder backend fault. The fix is to release the device inside the instance first, then let the detach complete — and only reset state if the attachment is genuinely stale.

Common Causes

• The volume’s filesystem is still mounted inside the guest (/mnt/data in use).
• A process has the device open (a database, a dd, an open file handle on the mount).
• The device is a member of an active LVM volume group, RAID array, or swap.
• A previous detach failed and left a stale attachment, so Cinder shows in-use/detaching while Nova/libvirt no longer has the device.
• The guest was hard-rebooted or crashed mid-detach, desyncing Nova and Cinder state.

Diagnostic Commands

Check the current volume and attachment state:

openstack volume show 4d9c1b2a-1234-5678-9abc-def012345678 -c status -c attachments
openstack server show my-instance -c volumes_attached

Inside the guest, find what is holding the device:

lsblk
mount | grep vdb
sudo lsof /dev/vdb
sudo fuser -vm /dev/vdb

On the compute host, confirm whether libvirt still has the disk attached:

sudo virsh domblklist <instance-name>
sudo journalctl -u nova-compute --since "-15min" | grep -i <volume-id>

Check the Cinder side for a stuck operation:

sudo tail -n 100 /var/log/cinder/cinder-volume.log | grep -i <volume-id>

Step-by-Step Resolution

1. Release the device inside the guest first. Stop any process using it, then unmount:

# in the instance:
sudo systemctl stop myapp
sudo umount /mnt/data

2. If umount still reports “target is busy,” find and stop the holder, then retry:

sudo fuser -km /mnt/data     # kills processes using the mount (use with care)
sudo umount /mnt/data

3. If the device is in an LVM group or RAID/swap, deactivate it before detaching:

sudo swapoff /dev/vdb        # if used as swap
sudo vgchange -an vg_data    # if part of an LVM VG

4. Now retry the detach from the control plane. With the guest no longer using the device, it should complete:

openstack server remove volume my-instance 4d9c1b2a-1234-5678-9abc-def012345678
openstack volume show 4d9c1b2a-... -c status

5. If the volume is stuck in detaching because of a stale attachment (libvirt no longer lists the disk, but Cinder still shows it attached), inspect and delete the orphaned attachment record:

openstack volume attachment list --volume 4d9c1b2a-...
openstack volume attachment delete <attachment-id>

6. As a last resort, when Nova and Cinder are desynced and you have confirmed libvirt does not have the device (virsh domblklist), reset the volume state so it returns to available. This changes bookkeeping only — do it only after verifying the dataplane is clear:

openstack volume set --state available 4d9c1b2a-1234-5678-9abc-def012345678
openstack volume show 4d9c1b2a-... -c status

Prevention

• Always unmount and stop consumers inside the guest before issuing a detach.
• Avoid hard-rebooting instances while a detach is in flight; it is the main source of stale attachments.
• Use consistent device names or UUIDs in /etc/fstab (and nofail) so a detached volume does not wedge the guest on next boot.
• Monitor for volumes lingering in detaching/attaching and alert; these are almost always desync artifacts.
• Deactivate LVM/RAID/swap on a volume as part of your decommission runbook before removing it.

Related Errors

• Volume status must be 'in-use' ... to detach (HTTP 400) — the control-plane rejection when the attachment state is inconsistent.
• Device vdb is in use — the libvirt/QEMU error that blocks the detach at the hypervisor.
• Volume ... did not finish being created — a Nova build fault when an attach never completes.
• cinder backup failed — a related Cinder operational failure worth checking if backups also break.

Frequently Asked Questions

Why can’t I just reset the state to available? Because volume set --state available only rewrites Cinder’s bookkeeping. If the guest is still using the device, you will corrupt data or leave the attachment truly orphaned. Release the device first, and reset state only when you have confirmed the dataplane is clear.

How do I know if the attachment is stale versus real? Run virsh domblklist on the compute host. If libvirt does not list the disk but Cinder shows in-use, the attachment is stale and safe to clean up.

Is this a Cinder backend bug? Rarely. The overwhelmingly common cause is the guest still holding the block device — a mount, an open process, or LVM/RAID/swap.

Can I get a safe detach runbook generated for my setup? Yes — describe your guest layout in the DevOps AI prompt library to produce a step-by-step detach and cleanup checklist.

Where can I find more Cinder troubleshooting? See the full OpenStack guides for related volume, attach, and backup fixes.