What actually breaks, and what to check first
Almost every painful Terraform incident is a *state* problem wearing a different costume. The configuration describes what you want, the provider describes what exists, and state is Terraform’s memory of the mapping between them. When those three disagree, Terraform does the only thing it can: it proposes to make reality match the configuration — which is why a corrupted or stale state file turns an innocuous plan into a destroy.
So the first question in any Terraform incident is never "what does the error say". It is "does state still describe reality". Answer that before you touch anything, because the recovery path for a lock is the opposite of the recovery path for drift, and applying the wrong one makes the other worse.
The second recurring theme is that Terraform errors are frequently *provider* errors in a Terraform wrapper. A 403 from AWS, a quota rejection from Azure, an eventual-consistency race in a cloud API — Terraform faithfully reports what it was told, and engineers spend an hour reading HCL when the problem was an IAM policy.
Triage order
Work down this list in order. Each step either finds the fault or rules out a whole class of cause — a negative result is progress, not a wasted step.
-
Is state locked by someone or something else?
terraform plan (read the lock error — it names the holder and lock ID)A lock error names who holds it and when they took it. A lock from a CI run that died is stale; a lock from a colleague mid-apply is not. Never force-unlock a lock you cannot account for — you can produce two concurrent applies against one state file.
-
Does state still match reality?
terraform plan -refresh-onlyThis shows drift and nothing else — what changed outside Terraform. An empty result means state is trustworthy and the problem is in your configuration. A large result means someone changed infrastructure by hand, and any normal plan you run will try to revert it.
-
Is the plan destructive, and is that intended?
terraform plan -out=tfplan && terraform show -json tfplan | jq '[.resource_changes[] | select(.change.actions[] | . == "delete")] | length'Counts resources the plan will delete. A non-zero count you did not expect is the single highest-value stop signal in Terraform. Replacements ("destroy and then create") are counted here too and are the usual cause of surprise downtime.
-
Is the error actually from the provider, not from Terraform?
TF_LOG=DEBUG terraform apply 2>&1 | grep -iE "status code|error response|denied|quota"Surfaces the raw cloud API response. If you see a 403, a quota rejection, or a throttle, the fix is in IAM, quotas or rate limits — not in your HCL. This one check reliably ends the longest Terraform debugging sessions.
-
Are provider and module versions pinned and consistent?
terraform version && terraform providersAn unpinned provider means CI and your laptop can resolve different versions, which produces the classic "works locally, fails in the pipeline" plan diff. Version constraints belong in `required_providers`, and `.terraform.lock.hcl` belongs in version control.
Diagnostic commands
Every command is labelled by what it can do to the system. Read-only commands are safe to run during an incident; the others are not, and are marked accordingly.
terraform plan -refresh-only Show only what changed outside Terraform, without proposing configuration changes.
How to read it: No changes = state is accurate. Any output = out-of-band drift you must reconcile before a normal plan is safe to trust.
terraform show -json tfplan | jq '.resource_changes[] | select(.change.actions[0] != "no-op") | {addr: .address, act: .change.actions}' List exactly which resources change, and how, from a saved plan file.
How to read it: `["delete"]` or `["delete","create"]` on anything stateful — a database, a volume, a load balancer address — should stop the apply.
terraform state list Enumerate everything Terraform believes it manages.
How to read it: A resource missing here that exists in the cloud will be *created again*, producing duplicates. A resource present here that no longer exists in the cloud will produce errors on refresh.
terraform state show <address> Inspect the recorded attributes of a single managed resource.
How to read it: Compare against the live resource. Differences here are the concrete content of "drift" and tell you whether the change was benign (a tag) or structural (a subnet).
terraform import <address> <cloud-id> Bring an existing, unmanaged resource under Terraform without recreating it.
How to read it: The correct answer to "Terraform wants to create something that already exists". Import first, then run a plan and expect it to be empty; a non-empty plan means your HCL does not yet match the real resource.
terraform force-unlock <LOCK_ID> Release a state lock left behind by a process that died.
How to read it: Only safe once you have positively confirmed no apply is still running. Two concurrent applies against one state file can corrupt it beyond automatic recovery.
terraform apply -target=<address> Apply a single resource, as an escape hatch during an incident.
How to read it: Leaves state and configuration partially converged. Acceptable to break a deadlock; always follow with a full untargeted plan and expect it to be empty.
Failure modes
These are distinct problems, not variations of one. Matching the symptom to the right cause is most of the work.
Error acquiring the state lock; the apply refuses to start.
- Cause:
- A previous run holds the lock — often a CI job that was cancelled or timed out before releasing it.
- Fix:
- Identify the holder from the lock metadata. If the process is genuinely gone, `terraform force-unlock <ID>`. If you cannot prove it is gone, wait — a corrupted state file costs far more than a delayed deploy.
A plan proposes to destroy and recreate a resource you did not change.
- Cause:
- A ForceNew attribute changed — frequently a provider upgrade altering a default, or a computed value that is not stable between runs.
- Fix:
- Read the plan’s `# forces replacement` annotations to find the attribute. If the change is unwanted, pin the provider version and set the attribute explicitly. `lifecycle { prevent_destroy = true }` on stateful resources turns this into an error instead of an outage.
Terraform tries to create a resource that already exists in the cloud.
- Cause:
- The resource exists but is not in state — created by hand, or lost when state was rebuilt or a workspace was switched.
- Fix:
- `terraform import` it to the correct address, then confirm with a plan that returns no changes. Creating it again produces duplicate infrastructure and, for named globals, a hard API conflict.
The plan differs between CI and a local machine with identical code.
- Cause:
- Different provider or module versions resolved, different workspace selected, or different variable files loaded.
- Fix:
- Commit `.terraform.lock.hcl`, pin `required_providers`, and print `terraform workspace show` in CI. Version drift is the usual answer; workspace confusion is the second.
Apply fails partway, leaving infrastructure half-built.
- Cause:
- A provider error mid-graph. Terraform records what succeeded, so state is accurate — but the deployment is incomplete.
- Fix:
- Do not delete state. Fix the underlying cause and re-run apply: Terraform will converge from where it is. Only use `-target` if a dependency ordering problem is blocking the whole graph.
"Provider produced inconsistent result after apply".
- Cause:
- The provider returned a value that differs from what it planned — usually an eventual-consistency race, occasionally a provider bug.
- Fix:
- Re-run apply; a converged second run confirms a race. If it persists, the resource attribute is genuinely unstable — check the provider’s issue tracker before working around it in HCL.
Common mistakes
- Running `terraform destroy` to "clean up" a broken environment that shares a state file with something in production.
- Deleting `terraform.tfstate` to escape an error. State is the only record of what Terraform manages; deleting it orphans every resource and the next apply recreates all of it.
- Leaving providers unpinned, then attributing the resulting plan diff to a code change.
- Committing `.tfvars` files containing credentials. Use environment variables or a secret manager; state itself can also contain sensitive values and belongs in an encrypted, access-controlled backend.
- Using `-target` routinely rather than as an incident escape hatch — it silently accumulates divergence between configuration and reality.
- Reviewing a plan by reading the summary line instead of the resource changes. "3 to add, 1 to change, 2 to destroy" does not tell you *which* two.
Frequently asked questions
How do I tell whether a Terraform error is my code or the cloud provider?
Run with `TF_LOG=DEBUG` and look for the raw API status code. Terraform surfaces provider responses more or less verbatim, so a 403, a quota rejection or a throttling response means the fix is in IAM, quotas or rate limiting rather than in your HCL. Configuration errors, by contrast, are usually caught at plan time and reference a file and line.
Is it ever safe to run terraform force-unlock?
Yes — when you can positively confirm the process holding the lock no longer exists, which most often means a CI job that was cancelled. The unsafe case is unlocking because you are impatient: two simultaneous applies writing one state file can corrupt it in ways that require restoring a backup of state.
What is the difference between drift and a normal plan diff?
Drift is reality changing underneath Terraform; a plan diff is your configuration changing on top of it. `terraform plan -refresh-only` isolates the first. This matters because the two have opposite fixes: drift is either imported or reverted, whereas a configuration diff is simply applied.
Should Terraform state live in version control?
No. State can contain secrets in plain text and has no locking mechanism in Git, which invites concurrent-apply corruption. Use a remote backend that supports locking and encryption. `.terraform.lock.hcl` is the opposite case — that one *should* be committed, because it is what makes provider resolution reproducible.
How do I stop Terraform from deleting a production database?
Put `lifecycle { prevent_destroy = true }` on the resource. It converts a destructive plan into an error, which is the behaviour you want when the alternative is silent data loss. Pair it with a CI gate that counts delete actions in the plan JSON so the intent is checked before a human reviews it.