☁️ OpenStack Error: Heat Stack Stuck in UPDATE_IN_PROGRESS and Cannot Update

Exact Error Message

$ openstack stack update app-cluster -t template.yaml
ERROR: Updating a stack when it is in status UPDATE_IN_PROGRESS is not
supported. The stack can be updated once it reaches a stable state
(UPDATE_COMPLETE, UPDATE_FAILED, CREATE_COMPLETE).

Looking at the stack, it has been “in progress” for far longer than the operation should take:

$ openstack stack show app-cluster -c stack_status -c updated_at
+---------------+----------------------------+
| Field         | Value                      |
+---------------+----------------------------+
| stack_status  | UPDATE_IN_PROGRESS         |
| updated_at    | 2026-07-17T02:14:06Z       |
+---------------+----------------------------+

What It Means

Heat treats UPDATE_IN_PROGRESS as an active, exclusive operation. While a stack is in this state, heat-engine believes a worker is still walking the resource graph, so it refuses any new update, and the API returns the error above. A stack becomes stuck in this state when the engine that owned the operation dies or loses its lock without ever writing a terminal status (UPDATE_COMPLETE or UPDATE_FAILED).

The two usual triggers are an engine crash/restart mid-update, and a resource whose “wait” never completes — a WaitCondition, a SoftwareDeployment, or a nested stack that Nova/Neutron never finishes provisioning. Because the state persists in the Heat database, the stack stays wedged until you cancel or reset it.

Common Causes

• heat-engine was restarted, crashed, or lost its RPC/messaging connection while owning the update.
• A OS::Heat::WaitCondition or SoftwareDeployment is waiting for a signal that never arrives (bad user-data, no metadata agent, blocked network).
• A downstream resource (Nova server, Neutron port, Cinder volume) is itself stuck, so the resource task never returns.
• A nested stack is in progress and the parent cannot advance past it.
• RabbitMQ/AMQP problems dropped the message that would have completed the task.
• The stack owner engine’s stale lock was never cleared after a controller reboot.

Diagnostic Commands

Confirm the stack status and how long it has been stuck:

openstack stack show app-cluster -c stack_status -c stack_status_reason -c updated_at

Find which resource the update is blocked on:

openstack stack resource list app-cluster --nested-depth 3 \
  -c resource_name -c resource_status -c resource_type

Check whether heat-engine is even running and holding the lock:

systemctl status heat-engine
journalctl -u heat-engine --since "1 hour ago" | grep -Ei 'lock|stack-id|traceback'

Look at the resource-level events to see where it stalled:

openstack stack event list app-cluster --nested-depth 3 --follow

Step-by-Step Resolution

1. Verify the engine is healthy before touching the stack. If heat-engine is down or flapping, restart it first — a returning engine sometimes reclaims and finishes the operation:

systemctl restart heat-engine

2. Identify the blocking resource from resource list. If it is a WaitCondition/SoftwareDeployment, the instance never signaled back — check the server console and metadata path rather than forcing Heat.

3. Ask Heat to cancel the running update cleanly. This is the safest first move and rolls the stack back to its last good state:

openstack stack cancel app-cluster
openstack stack show app-cluster -c stack_status

4. If cancel is refused because no live engine owns the operation, cancel without rollback so it lands in a terminal UPDATE_FAILED:

openstack stack cancel --no-rollback app-cluster

5. Once the stack reaches UPDATE_FAILED or UPDATE_COMPLETE, re-run your update:

openstack stack update app-cluster -t template.yaml

6. As a last resort for a truly orphaned stack (owner engine long gone, cancel does nothing), reset its state to UPDATE_FAILED with admin credentials, then update again. Only do this when you have confirmed no engine is still working the stack:

# Requires admin; forces the stack to a terminal state
heat-manage --config-file /etc/heat/heat.conf reset_stack_status app-cluster

Prevention

• Give WaitCondition/SoftwareDeployment resources explicit, realistic timeout values so a missed signal fails fast instead of hanging forever.
• Run multiple heat-engine workers and monitor them, so a single crash does not strand every in-flight stack.
• Alert on any stack in *_IN_PROGRESS for longer than an expected threshold (for example 30 minutes).
• Keep RabbitMQ/AMQP healthy and monitored — dropped Heat RPC messages are a classic cause of stuck tasks.
• Prefer openstack stack cancel (with rollback) over database surgery; reach for reset_stack_status only when the owner engine is confirmed dead.

Related Errors

• Updating a stack when it is in status CREATE_IN_PROGRESS is not supported — the same lock problem during initial create.
• WaitConditionTimeout: 0 of 1 received — the signal-never-arrived cause behind many stuck updates.
• Remote error: MessagingTimeout in heat-engine logs — AMQP dropped the completion message.
• ERROR: Resource DELETE failed when tearing down a wedged stack afterward.

Frequently Asked Questions

Can I just run the update again to unstick it? No — Heat rejects any new update while the stack is UPDATE_IN_PROGRESS. You must first move it to a terminal state with openstack stack cancel (optionally --no-rollback).

What is the difference between cancel and cancel —no-rollback? Plain cancel rolls the stack back to its previous good state; --no-rollback stops the operation and leaves the stack in UPDATE_FAILED at wherever it stalled, which is useful when no live engine can perform a rollback.

Why did the stack get stuck in the first place? Almost always a heat-engine that died mid-update, or a WaitCondition/SoftwareDeployment waiting on a signal from an instance that never sent one because of bad user-data or a broken metadata path.

Is it safe to reset the stack status manually? heat-manage reset_stack_status is safe only after you confirm no engine is still processing the stack; forcing the state while a worker is live can corrupt the resource graph.

Where can I find more OpenStack fixes? See the full OpenStack guides for Heat, Nova, and Neutron troubleshooting, and pull a reusable diagnostic prompt from the prompt library.