🚨 Terraform Error: Failed to Acquire State Lock in CI Automation Pipelines

Overview

Terraform takes a lock on its remote state before any plan or apply that could write, so two runs cannot corrupt the state file. In automation, a run that is canceled or killed mid-operation can leave that lock held. The next pipeline run then fails:

Error: Error acquiring the state lock

Error message: ConditionalCheckFailedException: The conditional request failed
Lock Info:
  ID:        9f2c1a44-8b6e-4d5a-bb0e-2f1c7a9e5d31
  Operation: OperationTypeApply
  Who:       runner@ci-agent-7
  Created:   2026-07-09 02:14:07.882 +0000 UTC

Every subsequent terraform apply in the pipeline fails to acquire the lock until it is released, blocking all deploys through that state.

Symptoms

• Error acquiring the state lock with a Lock Info block naming an operation, holder, and timestamp.
• The lock’s Created time lines up with a previously canceled or timed-out CI run.
• Local and CI runs against the same backend all block on the lock.
• With DynamoDB locking, ConditionalCheckFailedException accompanies the error.
• Pipelines serialize to a crawl or fail outright waiting on the lock.

Common Root Causes

• Canceled CI run — a job was canceled (concurrency cancellation, manual stop, timeout) between lock and unlock, so Terraform never released it.
• Killed process — the runner or container was terminated (OOM, spot reclaim) mid-apply.
• Concurrent runs on one state — two pipelines targeting the same state/workspace at once, one legitimately holding the lock.
• Backend/network blip — Terraform lost connectivity to the lock backend during release.
• Missing DynamoDB table or permissions — the lock table is misconfigured so locking is inconsistent.
• Force-unlock misuse — someone unlocked a lock that was actually still active, corrupting state.

Diagnostic Workflow

Read the full error — it prints who holds the lock and when it was created:

terraform apply 2>&1 | sed -n '/Lock Info/,/^$/p'

Decide whether the holder is a live run or a dead one. Check CI for an active job matching the holder/time:

gh run list --workflow terraform.yml --limit 10
# match the lock's Created time and 'Who' to a run; is that run still active?

For the S3 + DynamoDB backend, inspect the lock item directly:

aws dynamodb get-item \
  --table-name terraform-locks \
  --key '{"LockID":{"S":"my-bucket/env/prod/terraform.tfstate-md5"}}'

Confirm the backend and lock table configuration:

grep -RnA6 'backend "s3"' *.tf
aws dynamodb describe-table --table-name terraform-locks --query 'Table.TableStatus'

Only after confirming NO live run holds it, force-unlock with the exact lock ID:

terraform force-unlock 9f2c1a44-8b6e-4d5a-bb0e-2f1c7a9e5d31

Example Root Cause Analysis

A nightly deploy pipeline began failing every run with Error acquiring the state lock. The Lock Info showed Operation: OperationTypeApply, Who: runner@ci-agent-7, Created: 02:14:07. Cross-referencing gh run list showed a run at 02:13 that had been canceled by a concurrency rule at 02:14:20 — right in the middle of an apply. Terraform never got to release the lock.

The DynamoDB get-item confirmed the lock item still existed with that ID and no live process behind it. Because the holding run was verifiably dead (its GitHub job showed Cancelled), it was safe to release:

terraform force-unlock 9f2c1a44-8b6e-4d5a-bb0e-2f1c7a9e5d31
Terraform state has been successfully unlocked!

The next apply proceeded normally. The underlying cause was the same concurrency-cancellation pattern that had killed the earlier run, so the durable fix was to set cancel-in-progress: false on the deploy workflow so an in-flight apply is never interrupted, plus a -lock-timeout=5m on apply so a brief overlap waits instead of failing instantly.

Prevention Best Practices

• Never cancel an in-flight terraform apply in CI; set the deploy workflow’s cancel-in-progress: false so applies queue.
• Add -lock-timeout=<duration> (e.g. 5m) so a transient overlap waits for the lock instead of failing immediately.
• Serialize applies per state with a concurrency group keyed to the workspace/environment.
• Only force-unlock after proving the holding run is dead — match the lock’s Who/Created to a confirmed non-running job.
• Give applies enough runner memory and avoid spot interruption for the apply step so processes are not killed mid-lock.
• Verify the DynamoDB lock table (or backend lock mechanism) exists and the role has read/write on it.

Quick Command Reference

# See the lock holder details
terraform apply 2>&1 | sed -n '/Lock Info/,/^$/p'

# Is the holder a still-running job?
gh run list --workflow terraform.yml --limit 10

# Inspect the DynamoDB lock item
aws dynamodb get-item --table-name terraform-locks \
  --key '{"LockID":{"S":"<state-path>-md5"}}'

# Release ONLY after confirming the holder is dead
terraform force-unlock <LOCK_ID>

# Make future runs wait instead of failing
terraform apply -lock-timeout=5m

Conclusion

A Terraform state-lock timeout in CI is almost always a ghost: a previous run was killed or canceled between lock and unlock and never let go. The fix is two-sided. Immediately, identify the holder from the Lock Info block, confirm that run is truly dead, and force-unlock with the exact ID — never on a hunch, because unlocking a live apply corrupts state. Durably, stop interrupting applies (cancel-in-progress: false) and add a -lock-timeout so brief overlaps wait their turn instead of failing the pipeline.