🚨 OpenStack Error Guide: 'task_state deleting' — Instance Stuck Deleting in Nova

Overview

When you delete an OpenStack instance, Nova sets the instance’s task_state to deleting and hands the work to nova-compute, which must tear down the libvirt domain, detach volumes via Cinder, unplug ports via Neutron, and free resources. If any of those steps hangs — a Cinder call never returns, a Neutron port won’t clean up, or the RPC message to nova-compute is lost — the instance gets stuck with task_state pinned at deleting and never disappears.

The literal symptom, from openstack server show:

| status                | ACTIVE                               |
| OS-EXT-STS:task_state | deleting                             |
| OS-EXT-STS:vm_state   | active                               |

And in the nova_compute log you may see a wedged detach or unplug:

WARNING nova.compute.manager [instance: 6f2a...] Instance is being deleted but the deletion is taking longer than expected
ERROR nova.compute.manager ... Failed to detach volume ... Timeout waiting for device ... to be detached

The instance is not gone, but openstack server delete appears to have “worked” — the tell is a task_state of deleting that never clears. The delete is blocked on one of Nova’s dependencies (Cinder, Neutron, RabbitMQ, or libvirt), not on Nova’s own bookkeeping.

Symptoms

• openstack server delete <id> returns without error, but the instance stays listed.
• openstack server show <id> shows task_state = deleting for minutes/hours.
• nova_compute logs show a stuck volume detach, port unplug, or no activity at all.
• The backing volume stays in-use/detaching, or the Neutron port stays ACTIVE.

openstack server show <instance-id> -c status -c "OS-EXT-STS:task_state" -c "OS-EXT-STS:vm_state"

+-----------------------+-----------+
| Field                 | Value     |
+-----------------------+-----------+
| status                | ACTIVE    |
| OS-EXT-STS:task_state | deleting  |
| OS-EXT-STS:vm_state   | active    |
+-----------------------+-----------+

Common Root Causes

1. Stuck volume detach (Cinder)

The most common cause: Nova asks Cinder/libvirt to detach a volume, the guest never releases the device, and the detach times out — blocking the whole delete.

openstack server show <instance-id> -c volumes_attached
openstack volume list --status detaching
openstack volume show <volume-id> -c status -c attachments

| status      | detaching                                  |
| attachments | [{'server_id': '<instance-id>', ...}]      |

A volume stuck in detaching will hold the instance in deleting.

2. Neutron port cleanup failure

Nova must unplug and delete the instance’s ports. If neutron-server or an agent is unresponsive, the port never clears and the delete stalls.

openstack port list --device-id <instance-id>
docker logs nova_compute 2>&1 | grep -iE "unplug|neutron|port" | tail -5

ERROR nova.compute.manager ... Failed to unplug vif ... port <port-id> still bound

3. Lost RPC / RabbitMQ hiccup

The delete request travels as an oslo.messaging RPC to nova-compute. If RabbitMQ dropped the message or the compute agent was down when it was sent, the API set task_state=deleting but no worker ever picked it up.

openstack compute service list --service nova-compute -c Host -c State -c "Updated At"
docker logs nova_compute 2>&1 | grep -iE "MessagingTimeout|AMQP server|reply" | tail -5

| nova-compute | compute-02 | down |

If the compute host was down at delete time, the RPC was never consumed.

4. libvirt / hypervisor hang

If the underlying libvirt domain is wedged (D-state process, storage I/O hang), nova-compute can’t destroy it and the teardown blocks.

docker logs nova_compute 2>&1 | grep -iE "libvirt|destroy|domain" | tail -5
docker exec nova_libvirt virsh list --all | grep -i <instance-id>

5. Cinder or Neutron backend genuinely down

If the storage or network backend itself is unavailable, every detach/unplug call errors out and the delete cannot complete until the backend recovers.

openstack volume service list -c Binary -c Host -c State
openstack network agent list -c "Agent Type" -c Host -c Alive

Diagnostic Workflow

Step 1: Confirm the exact state (read-only)

openstack server show <instance-id> \
  -c status -c "OS-EXT-STS:task_state" -c "OS-EXT-STS:vm_state" \
  -c "OS-EXT-SRV-ATTR:host" -c volumes_attached

Note the compute host — you’ll read that host’s nova_compute log next.

Step 2: Read the nova-compute log on the instance’s host

docker logs nova_compute 2>&1 | grep -iE "$(echo <instance-id>)|detach|unplug|delete|Timeout" | tail -30

Look for the last thing Nova tried: a volume detach timeout, a vif unplug failure, or nothing at all (which points at a lost RPC).

Step 3: Check the attached volume and ports

openstack volume list --status detaching
openstack volume show <volume-id> -c status -c attachments
openstack port list --device-id <instance-id>

A volume in detaching or a lingering port names the specific blocked dependency.

Step 4: Verify the compute agent and message bus are healthy

openstack compute service list --service nova-compute -c Host -c State
docker exec rabbitmq rabbitmqctl list_queues name messages consumers \
  2>/dev/null | grep -i nova | sort -k2 -n -r | head

If nova-compute is down, or its queue has messages with 0 consumers, the delete RPC is not being processed — fix that first.

Step 5: Check backends and the hypervisor

openstack volume service list -c Binary -c Host -c State
openstack network agent list -c "Agent Type" -c Host -c Alive
docker exec nova_libvirt virsh list --all | grep -i <instance-id>

Example Root Cause Analysis

An operator deletes instance 6f2a1b3c-... but it stays listed. openstack server show reports task_state = deleting, vm_state = active, host compute-02. Reading that host’s compute log:

docker logs nova_compute 2>&1 | grep -iE "detach|Timeout" | tail -5

ERROR nova.virt.libvirt.driver [instance: 6f2a...] Waiting for libvirt to detach device vdb timed out
ERROR nova.compute.manager [instance: 6f2a...] Failed to detach volume <volume-id> from /dev/vdb

The delete is blocked on a volume detach. Confirming with Cinder:

openstack volume show <volume-id> -c status -c attachments

| status | detaching |

The guest OS never released /dev/vdb (a common result of a hung guest or an in-flight fsck), so libvirt’s detach timed out and Nova gave up mid-delete. The message bus and agents are healthy, so this is a genuine detach hang, not a lost RPC.

Because the instance is truly wedged and read-only checks confirm the cause, the last-resort recovery is to reset the task_state and force the delete — with cautions. First clear the stuck task_state so Nova will accept a new operation, then force-delete:

# Clears task_state back to None (keeps the instance active); does NOT touch the volume
nova reset-state --active <instance-id>

# Force-delete bypasses the normal graceful teardown
openstack server delete --force <instance-id>

Cautions: --force skips the graceful detach, so the Cinder volume can be left in detaching/in-use with a stale attachment. After forcing the delete, verify and, if needed, clear the volume’s state (an admin action) so it can be reused:

openstack volume show <volume-id> -c status -c attachments
# If still stuck attached with no instance:
cinder reset-state --state available --attach-status detached <volume-id>

Only clear volume state after confirming no instance is actually using it — resetting a genuinely-attached volume risks data corruption. Similarly, never force-delete blindly across many instances; each one may be blocked on a different dependency you’d be masking.

Prevention Best Practices

• Monitor task_state dwell time: alert when any instance sits in deleting (or building) longer than a few minutes so you catch wedged teardowns early.
• Keep RabbitMQ and nova-compute healthy — most “lost delete” cases are really a message-bus or down-agent problem. See the MessagingTimeout guide for the RPC side.
• Watch for volumes stuck in detaching/attaching; a Cinder backend that’s slow to detach is the top cause of stuck deletes.
• Prefer fixing the blocked dependency (restart the specific down nova_compute, recover the Cinder/Neutron backend) before reaching for reset-state/--force.
• Treat nova reset-state and server delete --force as last resorts: they bypass safety and can orphan volumes and ports. Always audit and clean up the volume/port afterward.
• Never bulk --force delete stuck instances; investigate one, since a shared backend outage often explains all of them at once.

Quick Command Reference

# Confirm the stuck state and the compute host (read-only)
openstack server show <instance-id> \
  -c status -c "OS-EXT-STS:task_state" -c "OS-EXT-STS:vm_state" \
  -c "OS-EXT-SRV-ATTR:host" -c volumes_attached

# What did nova-compute last try?
docker logs nova_compute 2>&1 | grep -iE "detach|unplug|delete|Timeout" | tail -30

# Blocked volume / port
openstack volume list --status detaching
openstack volume show <volume-id> -c status -c attachments
openstack port list --device-id <instance-id>

# Is the agent / bus healthy?
openstack compute service list --service nova-compute -c Host -c State
docker exec rabbitmq rabbitmqctl list_queues name messages consumers | grep -i nova

# Backends and hypervisor
openstack volume service list -c Binary -c Host -c State
openstack network agent list -c "Agent Type" -c Host -c Alive
docker exec nova_libvirt virsh list --all | grep -i <instance-id>

# LAST RESORT (after confirming cause) — reset state then force delete
nova reset-state --active <instance-id>
openstack server delete --force <instance-id>
# Then clean up an orphaned volume ONLY if truly unattached:
cinder reset-state --state available --attach-status detached <volume-id>

Conclusion

An instance stuck in task_state deleting means Nova started the teardown but a dependency never let it finish — the delete is blocked, not lost in Nova’s own state. The diagnostic signature is a task_state of deleting that never clears while vm_state stays active. Typical root causes:

1. A stuck volume detach in Cinder/libvirt (detaching that never completes).
2. A Neutron port that won’t unplug or delete.
3. A lost RPC — RabbitMQ dropped the message or nova-compute was down.
4. A libvirt/hypervisor hang preventing domain destroy.
5. A Cinder or Neutron backend that’s genuinely unavailable.

Read the instance state and the nova_compute log first to identify the blocked step, fix that dependency, and only reach for nova reset-state --active and openstack server delete --force as a last resort — then clean up any orphaned volume or port they leave behind.