Terraform Backend State Lock Timeout Tuning Prompt

You are a senior Terraform engineer who has run state locking at scale across dozens of pipelines — you know exactly when a lock is protecting you and when it is just a self-inflicted queue.

I will provide:
- Backend type (S3+DynamoDB, S3 native lockfile, Terraform Cloud, GCS, azurerm)
- How many pipelines/engineers hit the same state
- The lock error(s) or symptoms
- Current backend config and CI concurrency settings

Your job:

1. **Diagnose the contention**, distinguish:
   - **Healthy serialization** — two applies on one state; the lock is doing its job.
   - **Self-contention** — one pipeline's plan and apply stages fighting, or matrix jobs sharing a state key.
   - **Stale lock** — a crashed run left a lock nobody holds.
   - **Backend throttling** — DynamoDB `ProvisionedThroughputExceeded`, not a real lock at all.
2. **Tune `-lock-timeout`** deliberately:
   - Explain what it actually does (retry-until, not force).
   - Recommend a value based on typical apply duration, not a round number.
   - Warn where a long timeout just hides a design problem (shared state key).
3. **State layout** — when the real fix is splitting one giant state into smaller ones (per-service, per-env) so runs stop colliding.
4. **CI concurrency** — map the fix to the platform: GitHub Actions `concurrency` groups, GitLab `resource_group`, TFC run queue. The goal is to serialize at the pipeline layer so Terraform never has to.
5. **Backend-specific tuning**:
   - S3+DynamoDB: throughput/on-demand, and why throttling looks like a lock.
   - S3 native lockfile (`use_lockfile`): trade-offs vs DynamoDB.
   - TFC/GCS/azurerm: their equivalent knobs.
6. **Safe recovery** — how to inspect a lock and when `force-unlock <ID>` is safe (only after proving no run holds it).

Give concrete config, the reasoning, and a way to verify the contention is gone (not just masked by a bigger timeout).

Mark DESTRUCTIVE: `terraform force-unlock` while another apply is genuinely running (state corruption), and disabling locking (`-lock=false`) on shared state.

---

Backend: [DESCRIBE]
Concurrency: [pipelines / engineers on this state]
Lock symptoms: [PASTE errors]
Current config: [PASTE backend + CI concurrency]

Why this prompt works

Lock errors get “fixed” by bumping the timeout until the queue is invisible, not gone. This prompt separates a lock that is protecting you from a lock that is a symptom of a bad state layout or CI concurrency, then fixes the right layer.

How to use it

1. Reproduce the lock error and capture the lock ID/holder.
2. Classify contention before changing any number.
3. Serialize at the pipeline layer so Terraform rarely queues.
4. Only then tune -lock-timeout for the residual case.

Useful commands

# Retry acquiring the lock for up to 5 minutes instead of failing instantly
terraform apply -lock-timeout=5m

# Inspect who/what holds a lock (error output includes ID, Who, Created)
terraform plan   # read the LockInfo block on failure

# Release a stale lock — ONLY after proving no run holds it
terraform force-unlock <LOCK_ID>

Read the lock, do not guess

Error: Error acquiring the state lock

Lock Info:
  ID:        3f2b9c10-...-a1
  Path:      s3://acme-tfstate/prod/network.tfstate
  Operation: OperationTypeApply
  Who:       runner@ci-42
  Created:   2026-07-06 14:03:11 UTC

If Who/Created point at a run that already finished, it is a stale lock. If they point at a run still executing, the lock is healthy — wait, do not force it.

Serialize at the pipeline layer

# GitHub Actions: one concurrent run per state, queue the rest
concurrency:
  group: terraform-${{ github.workflow }}-prod-network
  cancel-in-progress: false

# GitLab CI: resource_group serializes jobs on shared state
apply:prod:
  resource_group: prod-network-state
  script:
    - terraform apply -lock-timeout=3m -auto-approve tfplan

Backend-specific tuning

# S3 + DynamoDB: on-demand billing so locking never throttles
resource "aws_dynamodb_table" "tf_lock" {
  name         = "terraform-locks"
  billing_mode = "PAY_PER_REQUEST"   # avoids ProvisionedThroughputExceeded
  hash_key     = "LockID"
  attribute {
    name = "LockID"
    type = "S"
  }
}

# S3 native lockfile (no DynamoDB) — simpler, fewer moving parts
terraform {
  backend "s3" {
    bucket       = "acme-tfstate"
    key          = "prod/network.tfstate"
    region       = "us-east-1"
    use_lockfile = true
  }
}

Choosing a lock-timeout

apply p95 duration   suggested -lock-timeout   note
------------------   -----------------------   ----------------------------------
< 1 min              1m                         Short queues, fail fast
1-5 min              5m                          Cover one in-flight apply
> 5 min              re-evaluate state layout    A long queue = split the state
matrix / fan-out     serialize in CI instead     Do not let jobs share one key

Common findings this catches

• Matrix jobs sharing one state key → give each a distinct backend key.
• Plan+apply stages deadlocking one pipeline → single concurrency group.
• DynamoDB throttling read as lock contention → switch to on-demand.
• Stale locks after cancelled runs → verify holder, then force-unlock.
• One monolithic state everyone queues on → split per service.

When to escalate

• Repeated stale locks → runners are being killed mid-apply; fix the runner.
• State corruption suspected after a forced unlock → restore from versioned backend.
• Contention that only state-splitting fixes → plan a moved/state mv migration.