🚨 CloudFormation Error Guide: 'UPDATE_ROLLBACK_FAILED' — Recover a Stuck Stack

Overview

UPDATE_ROLLBACK_FAILED is one of the few CloudFormation stack states that blocks all further work: while the stack sits in this state you cannot run a normal update-stack or delete-stack. It happens when a stack update fails, CloudFormation automatically tries to roll back to the last known-good state, and the rollback itself fails on one or more resources.

You will see it in the console or from the CLI as the terminal status of a failed update:

Stack arn:aws:cloudformation:us-east-1:111122223333:stack/payments-prod/abcd...
is in UPDATE_ROLLBACK_FAILED state and can not be updated.

The stack event that caused it usually reads like this:

UPDATE_ROLLBACK_FAILED  AWS::RDS::DBInstance  payments-db
The following resource(s) failed to update: [payments-db]. Rollback requested by user.

The key idea: the stack is not broken beyond repair, but CloudFormation needs you to resolve whatever blocked the rollback (a resource it cannot restore) before it will move the stack back to UPDATE_COMPLETE or UPDATE_ROLLBACK_COMPLETE. You do that with continue-update-rollback, optionally skipping the resources that cannot roll back.

Symptoms

• aws cloudformation update-stack returns ... is in UPDATE_ROLLBACK_FAILED state and can not be updated.
• delete-stack refuses or leaves the stack stuck.
• The console shows the stack in orange/red UPDATE_ROLLBACK_FAILED with one or more resources flagged.
• A specific resource (often RDS, a custom resource, an ASG, or a security group) failed during the rollback phase, not the original update.

aws cloudformation describe-stacks \
  --stack-name payments-prod \
  --query 'Stacks[0].StackStatus'

"UPDATE_ROLLBACK_FAILED"

Common Root Causes

1. A resource cannot be restored to its previous state

The most common cause. During rollback CloudFormation tries to return a resource to its pre-update configuration, but the provider rejects it — for example an RDS instance that cannot be resized back because of storage or engine constraints, or a resource that was modified out-of-band.

aws cloudformation describe-stack-events \
  --stack-name payments-prod \
  --query 'StackEvents[?ResourceStatus==`UPDATE_ROLLBACK_FAILED`].[LogicalResourceId,ResourceStatusReason]' \
  --output table

------------------------------------------------------------
|  payments-db  |  Cannot modify DB instance: storage size  |
|               |  cannot be reduced.                        |
------------------------------------------------------------

2. A custom resource never sent a response

A Lambda-backed custom resource that failed, timed out, or did not signal back to the pre-signed S3 URL leaves CloudFormation waiting until the rollback times out and fails.

aws cloudformation describe-stack-events \
  --stack-name payments-prod \
  --query 'StackEvents[?contains(ResourceType,`Custom`)].[LogicalResourceId,ResourceStatusReason]' \
  --output table

-----------------------------------------------------------------
|  DbMigration  |  Failed to receive response from custom        |
|               |  resource. See CloudWatch logs.               |
-----------------------------------------------------------------

3. Out-of-band changes broke CloudFormation’s assumptions

Someone changed a resource manually (console, CLI, another tool) after CloudFormation created it. When the rollback tries to restore the resource, the real object no longer matches what CloudFormation expects and the API call fails.

4. A dependency was deleted or is in a bad state

A security group, subnet, IAM role, or target group that a rolling-back resource depends on was deleted or is itself in a failed state, so the restore cannot complete.

5. Insufficient permissions during rollback

The role CloudFormation uses lost a permission (a policy was tightened mid-flight), so it can update the resource forward but not restore it.

Resource handler returned message: "User: arn:aws:sts::111122223333:assumed-role/cfn-exec
is not authorized to perform: rds:ModifyDBInstance"

Diagnostic Workflow

Step 1: Confirm the state and find the blocking resources

aws cloudformation describe-stacks \
  --stack-name payments-prod \
  --query 'Stacks[0].StackStatus'

aws cloudformation describe-stack-events \
  --stack-name payments-prod \
  --max-items 40 \
  --query 'StackEvents[?ResourceStatus==`UPDATE_ROLLBACK_FAILED`].[LogicalResourceId,ResourceType,ResourceStatusReason]' \
  --output table

Every logical ID that appears here is a candidate to fix or skip.

Step 2: Read the reason, not just the status

The ResourceStatusReason is the whole story — “storage size cannot be reduced”, “no response from custom resource”, “not authorized”. Fix the underlying cause where you can; if the resource genuinely cannot be rolled back, plan to skip it in Step 4.

Step 3: Fix the underlying problem where possible

For a custom resource, check its logs and, if needed, manually signal completion so CloudFormation stops waiting:

aws logs tail /aws/lambda/db-migration --since 30m

# If the resource is safe to consider complete, send SUCCESS to its callback URL:
curl -X PUT -H 'Content-Type:' \
  -d '{"Status":"SUCCESS","PhysicalResourceId":"db-migration","StackId":"...","RequestId":"...","LogicalResourceId":"DbMigration"}' \
  "https://cloudformation-custom-resource-response-...s3.amazonaws.com/..."

For a permissions cause, restore the missing permission on the execution role before continuing.

Step 4: Continue the rollback, skipping only what cannot recover

Once fixable causes are addressed, resume the rollback. For resources that genuinely cannot roll back, list them in --resources-to-skip (they will be left in their current state and marked UPDATE_COMPLETE):

# Try a clean continue first:
aws cloudformation continue-update-rollback \
  --stack-name payments-prod

# If specific resources still block it, skip them explicitly:
aws cloudformation continue-update-rollback \
  --stack-name payments-prod \
  --resources-to-skip payments-db DbMigration

For a resource nested in a child stack, qualify the name as NestedStackLogicalId.ResourceLogicalId.

Step 5: Verify the stack reached a stable state

aws cloudformation describe-stacks \
  --stack-name payments-prod \
  --query 'Stacks[0].StackStatus'

"UPDATE_ROLLBACK_COMPLETE"

UPDATE_ROLLBACK_COMPLETE means the stack is deployable again. Reconcile any skipped resource so template and reality match before the next update.

Example Root Cause Analysis

A deploy to payments-prod changed the RDS instance class and reduced AllocatedStorage in the same update. The forward update failed, CloudFormation began rolling back, and the rollback stalled at UPDATE_ROLLBACK_FAILED on payments-db.

Reading the events revealed the cause:

aws cloudformation describe-stack-events \
  --stack-name payments-prod \
  --query 'StackEvents[?LogicalResourceId==`payments-db`].[ResourceStatus,ResourceStatusReason]' \
  --output table

-------------------------------------------------------------------
|  UPDATE_ROLLBACK_FAILED  |  Cannot modify DB instance: storage   |
|                          |  size cannot be reduced.              |
-------------------------------------------------------------------

RDS storage can only grow, never shrink. The forward change had already grown storage; the rollback tried to shrink it back and RDS refused. The database itself was healthy and serving traffic — only CloudFormation’s bookkeeping was stuck.

The safe recovery was to skip the RDS resource in the rollback (leaving the live database untouched) and then reconcile the template:

aws cloudformation continue-update-rollback \
  --stack-name payments-prod \
  --resources-to-skip payments-db

{ }   # returns empty on success; poll status next

The stack moved to UPDATE_ROLLBACK_COMPLETE. The template was then updated to reflect the actual (larger) storage size so the next deploy showed no drift, and the storage change was reverted separately through a supported path.

Prevention Best Practices

• Never combine a risky, hard-to-reverse change (RDS resize, engine upgrade, storage change) with unrelated changes in one update — isolate it so a rollback has a clear path. See our infrastructure-as-code guides.
• Review a change set with describe-change-set before executing; replacements and modifications to stateful resources are exactly the ones that strand rollbacks.
• Make custom resources robust: always send a response (even on failure), set sensible timeouts, and log to CloudWatch so a stuck rollback is diagnosable.
• Treat CloudFormation-managed resources as read-only in the console; out-of-band edits are a leading cause of rollback failures.
• Enable termination protection and DeletionPolicy: Retain / UpdateReplacePolicy: Retain on stateful resources so a skip never risks data.
• When triaging a stuck stack under pressure, the free incident assistant can summarize the failing events into the likely blocking resource.

Quick Command Reference

# Current stack status
aws cloudformation describe-stacks --stack-name STACK \
  --query 'Stacks[0].StackStatus'

# Resources that failed the rollback, with reasons
aws cloudformation describe-stack-events --stack-name STACK \
  --query 'StackEvents[?ResourceStatus==`UPDATE_ROLLBACK_FAILED`].[LogicalResourceId,ResourceStatusReason]' \
  --output table

# Resume the rollback cleanly
aws cloudformation continue-update-rollback --stack-name STACK

# Resume, skipping resources that cannot roll back
aws cloudformation continue-update-rollback --stack-name STACK \
  --resources-to-skip LOGICAL_ID_A LOGICAL_ID_B

# Skip a resource inside a nested stack
#   --resources-to-skip NestedStackLogicalId.ResourceLogicalId

# Custom-resource Lambda logs
aws logs tail /aws/lambda/FUNCTION --since 30m

Conclusion

UPDATE_ROLLBACK_FAILED means a stack update failed and the automatic rollback then failed too, leaving the stack locked. The usual root causes:

1. A resource (often RDS) cannot be restored to its previous configuration.
2. A custom resource never signaled back and the rollback timed out.
3. Out-of-band changes broke CloudFormation’s assumptions during restore.
4. A deleted or unhealthy dependency blocked the rollback.
5. The execution role lost a permission needed to restore a resource.

Read the ResourceStatusReason for each failed resource, fix what you can, then run continue-update-rollback — using --resources-to-skip only for resources that genuinely cannot roll back — until the stack reaches UPDATE_ROLLBACK_COMPLETE. Then reconcile the template with reality so the next deploy is clean.