🚨 Terraform Error Guide: 'state data in S3 does not have the expected content' — Reconcile the DynamoDB Digest

Overview

The S3 backend can use a DynamoDB table not only for locking but also for consistency checking. After each successful state write, Terraform stores an MD5 digest of the state object in a DynamoDB item whose LockID ends in -md5. On the next operation it fetches the S3 object, recomputes the digest, and compares it to the stored one. When the two disagree, Terraform refuses to load state rather than risk operating on a stale or partially-written object:

Error: Error loading state:
    state data in S3 does not have the expected content.

This may be caused by unusually long delays in S3 processing a previous state
update. Please wait for a minute or two and try again. If this problem
persists, and neither S3 nor DynamoDB are experiencing an outage, you may need
to manually verify the remote state and update the Digest value stored in the
DynamoDB table to the correct value.

The message itself hints at both the benign cause (S3 read-after-write lag) and the real one (the DynamoDB digest no longer matches the actual S3 object). The check is a guardrail: the S3 object and DynamoDB’s record of it have drifted, and Terraform will not proceed until they agree again.

Symptoms

• terraform plan / apply / state commands fail on load with state data in S3 does not have the expected content.
• It often appears right after a previous apply was interrupted (network drop, CI timeout, Ctrl-C) or after someone edited state.
• Retrying a minute later sometimes works (genuine S3 eventual-consistency lag) but often does not (a real digest mismatch).
• The S3 state object exists and looks valid when downloaded, yet Terraform still rejects it.
• The DynamoDB lock table has a LockID row ending in -md5 whose Digest value does not match the current object’s MD5.

Common Root Causes

• Interrupted state write — an apply was killed after S3 was updated but before the DynamoDB digest was refreshed (or vice versa), leaving the two out of sync.
• Manual state edits — someone edited the state in S3 directly, or ran aws s3 cp to restore an older/newer version, without updating the digest row.
• S3 versioning rollback — reverting the object to a previous version changes its MD5 but leaves the stale digest in DynamoDB.
• Genuine S3 processing delay — a real but transient read-after-write lag, which the retry advice addresses.
• Concurrent/duplicate writers — two runs racing on the same key so the object written last does not match the digest recorded.
• Digest row corruption or partial DynamoDB write — the -md5 item was written incompletely or deleted and recreated wrong.

Diagnostic Workflow

First, take the message’s own advice and wait — a true S3 lag clears on its own:

sleep 120
terraform plan     # retry once; if it succeeds it was transient S3 lag

If it persists, compare the actual S3 object digest against what DynamoDB has stored. Identify your backend keys, then compute both digests:

# backend.tf -- the values you need for the checks below
terraform {
  backend "s3" {
    bucket         = "acme-tfstate"
    key            = "prod/network/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "acme-tf-locks"
  }
}

Download the current state and compute its MD5 (this is the digest Terraform expects):

aws s3 cp s3://acme-tfstate/prod/network/terraform.tfstate ./state.json
md5sum ./state.json     # note the hex digest

# Read the digest DynamoDB currently holds for this object.
# The LockID is "<bucket>/<key>-md5".
aws dynamodb get-item \
  --table-name acme-tf-locks \
  --key '{"LockID":{"S":"acme-tfstate/prod/network/terraform.tfstate-md5"}}'

If the Digest returned by DynamoDB differs from the md5sum of the object, that is the mismatch. Confirm the object itself is the state you actually want (check serial and lineage) before repairing:

jq '{serial, lineage, terraform_version}' ./state.json
aws s3api list-object-versions --bucket acme-tfstate \
  --prefix prod/network/terraform.tfstate      # if versioning is on

Example Root Cause Analysis

A CI pipeline running terraform apply was cancelled mid-run when the build job hit its wall-clock limit. The apply had already uploaded the new state object to S3, but the process died before Terraform updated the DynamoDB -md5 digest. The next pipeline run failed on load with state data in S3 does not have the expected content.

Waiting and retrying did not help — this was not S3 lag but a real drift. The engineer computed the S3 object’s MD5 and read the DynamoDB digest:

aws s3 cp s3://acme-tfstate/prod/network/terraform.tfstate ./state.json
md5sum ./state.json
# 7f3b2c9a...   <- actual object digest

aws dynamodb get-item --table-name acme-tf-locks \
  --key '{"LockID":{"S":"acme-tfstate/prod/network/terraform.tfstate-md5"}}'
# Digest -> 1a9e4d02...  (stale, from the pre-interrupt object)

The object was confirmed to be the correct, complete post-apply state (serial had advanced as expected). Because the S3 object was authoritative and only the digest was stale, the safe repair was to update the DynamoDB digest to match the current object:

aws dynamodb put-item \
  --table-name acme-tf-locks \
  --item '{"LockID":{"S":"acme-tfstate/prod/network/terraform.tfstate-md5"},
           "Digest":{"S":"7f3b2c9a..."}}'

The next terraform plan loaded cleanly. (Had the S3 object instead been the wrong one, the correct move would have been to restore the intended object version from S3 versioning first, then set the digest to that object’s MD5 — never the reverse.)

Prevention Best Practices

• Enable S3 bucket versioning on the state bucket so you can always recover the correct object version before repairing a digest.
• Never interrupt apply mid-write; give CI jobs generous timeouts and avoid Ctrl-C during state operations. If a run is killed, expect to reconcile before the next run.
• Avoid manual edits to remote state; use terraform state subcommands, and if you must touch S3 directly, update the DynamoDB -md5 digest in the same change.
• Keep the DynamoDB lock table (dynamodb_table, or the newer use_lockfile S3 native locking) enabled so concurrent runs cannot race and produce mismatches.
• Back up state before risky operations with terraform state pull > backup.tfstate so you have a known-good reference digest.
• Serialize pipelines per state key so two jobs never write the same object concurrently.

Quick Command Reference

# 1. Try the benign fix first: wait out possible S3 lag
sleep 120 && terraform plan

# 2. Pull a safe backup of remote state
terraform state pull > backup.tfstate

# 3. Compute the actual S3 object digest
aws s3 cp s3://acme-tfstate/prod/network/terraform.tfstate ./state.json
md5sum ./state.json

# 4. Read the digest DynamoDB currently stores
aws dynamodb get-item --table-name acme-tf-locks \
  --key '{"LockID":{"S":"acme-tfstate/prod/network/terraform.tfstate-md5"}}'

# 5. If the S3 object is authoritative, set the digest to match it
aws dynamodb put-item --table-name acme-tf-locks \
  --item '{"LockID":{"S":"acme-tfstate/prod/network/terraform.tfstate-md5"},
           "Digest":{"S":"<md5-of-state.json>"}}'

# 6. Inspect object versions if versioning is enabled
aws s3api list-object-versions --bucket acme-tfstate \
  --prefix prod/network/terraform.tfstate

# 7. Clear a stuck lock only after verifying no run is active
terraform force-unlock <LOCK_ID>

Conclusion

state data in S3 does not have the expected content is Terraform’s consistency guardrail firing: the MD5 digest stored in DynamoDB no longer matches the state object in S3. Start with the message’s own advice — wait a minute and retry — because sometimes it is only S3 read-after-write lag. When it persists, compare the object’s md5sum against the DynamoDB -md5 digest, confirm which artifact is authoritative (usually the S3 object, verified by serial/lineage), and reconcile the digest to match it. Enable bucket versioning, avoid interrupted applies and manual edits, and serialize pipelines so the digest and object never drift in the first place.