🚨 Terraform Error Guide: 'Failed to get existing workspaces' — Fix Backend Credentials, IAM, and Bucket Config

Overview

Failed to get existing workspaces is a backend error. It fires during terraform init, terraform workspace list, or any command that first enumerates the workspaces stored in your remote backend. For the S3 backend, Terraform lists the objects under your state key prefix to discover which workspaces exist; if that ListObjects/ListBucket call is denied, the bucket is wrong, the region is mismatched, or the credentials are missing or expired, Terraform cannot build the workspace list and stops before doing anything else.

The literal message, with the underlying provider error appended, looks like this:

Error: Failed to get existing workspaces: S3 bucket "my-tfstate-bucket" does not exist.

The referenced S3 bucket must have been previously created. If the S3 bucket
was created within the last minute, please wait for a minute or two and try
again.

A permissions variant surfaces the raw AWS API error:

Error: Failed to get existing workspaces: operation error S3: ListObjectsV2,
https response error StatusCode: 403, ... AccessDenied: Access Denied

The text after the colon is the real cause — an AccessDenied, a NoSuchBucket, an InvalidAccessKeyId, or a region redirect. Always read past Failed to get existing workspaces: to the appended provider message.

Symptoms

• terraform init fails at the “Initializing the backend…” step with Failed to get existing workspaces.
• terraform workspace list and terraform plan fail the same way.
• The trailing error is one of: AccessDenied (403), NoSuchBucket, InvalidAccessKeyId, ExpiredToken, SignatureDoesNotMatch, or a 301/PermanentRedirect (region mismatch).
• It works for one engineer but not another — a sign of per-user credentials or IAM policy differences.
• CI suddenly fails after a role, bucket, or region change even though the .tf files did not change.

Common Root Causes

• Missing s3:ListBucket: the IAM principal can GetObject/PutObject but lacks ListBucket on the bucket ARN — Terraform must list to enumerate workspaces.
• Wrong bucket name or it does not exist: a typo in bucket, a bucket in another account, or a bucket never created.
• Region mismatch: the region in the backend block does not match the bucket’s actual region, producing a PermanentRedirect/301.
• Missing, wrong, or expired credentials: no AWS_PROFILE/AWS_ACCESS_KEY_ID, an expired STS session token, or the wrong profile assumed.
• DynamoDB lock table permissions: with dynamodb_table set, the principal needs dynamodb:GetItem/PutItem/DeleteItem (and the table must exist) or related calls fail.
• Bucket policy or SCP denial: an explicit Deny in the bucket policy, an Organizations SCP, or a VPC endpoint policy blocking s3:ListBucket.
• KMS access missing: if state objects are SSE-KMS encrypted, missing kms:Decrypt can surface as access errors when Terraform reads existing state.

Diagnostic Workflow

Confirm which identity Terraform is actually using, then test the exact S3 calls it makes:

# Who am I, per the same credential chain Terraform uses?
aws sts get-caller-identity

# Does the bucket exist and what region is it really in?
aws s3api get-bucket-location --bucket my-tfstate-bucket

# Can this identity LIST the state prefix (the call Terraform makes)?
aws s3api list-objects-v2 --bucket my-tfstate-bucket --prefix env: --max-items 5

# If a lock table is configured, can I reach it?
aws dynamodb describe-table --table-name my-tf-locks --region us-east-1

Re-run init with debug logging to see the raw provider error:

TF_LOG=DEBUG terraform init 2>&1 | grep -i 'ListObjects\|AccessDenied\|region\|bucket'

Here is a backend configuration that triggers the error — the region is wrong for the bucket:

terraform {
  backend "s3" {
    bucket         = "my-tfstate-bucket"
    key            = "prod/terraform.tfstate"
    region         = "us-west-2"       # bucket actually lives in us-east-1
    dynamodb_table = "my-tf-locks"
    encrypt        = true
  }
}

And the minimal IAM policy Terraform needs — note ListBucket is on the bucket ARN, object actions on the object ARN:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::my-tfstate-bucket" },
    { "Effect": "Allow", "Action": ["s3:GetObject","s3:PutObject","s3:DeleteObject"],
      "Resource": "arn:aws:s3:::my-tfstate-bucket/*" },
    { "Effect": "Allow", "Action": ["dynamodb:GetItem","dynamodb:PutItem","dynamodb:DeleteItem"],
      "Resource": "arn:aws:dynamodb:us-east-1:111122223333:table/my-tf-locks" }
  ]
}

Example Root Cause Analysis

A pipeline that had worked for months began failing every terraform init with:

Error: Failed to get existing workspaces: operation error S3: ListObjectsV2,
https response error StatusCode: 403, ... AccessDenied: Access Denied

aws sts get-caller-identity inside the runner showed the CI role was assuming correctly, and aws s3api get-bucket-location returned the right region — so credentials and region were fine. But aws s3api list-objects-v2 --bucket my-tfstate-bucket returned AccessDenied, while aws s3api get-object on a specific key succeeded. That asymmetry pinned it: the role’s policy granted s3:GetObject/s3:PutObject on arn:aws:s3:::my-tfstate-bucket/* but had no s3:ListBucket statement on arn:aws:s3:::my-tfstate-bucket. A recent policy refactor had dropped the bucket-level statement. Adding the s3:ListBucket statement (on the bucket ARN, not the object ARN) restored terraform init. Root cause: object-level permissions without the bucket-level ListBucket that workspace enumeration requires.

Prevention Best Practices

• Attach the full backend IAM policy — s3:ListBucket on the bucket ARN plus object actions on /* — and keep it in code so refactors cannot silently drop a statement.
• Pin the backend region to the bucket’s real region; verify with aws s3api get-bucket-location when creating the bucket.
• Add the dynamodb:GetItem/PutItem/DeleteItem permissions and confirm the lock table exists before enabling dynamodb_table.
• If state is SSE-KMS encrypted, grant kms:Encrypt/kms:Decrypt/kms:GenerateDataKey on the key.
• Validate credentials in CI with aws sts get-caller-identity before terraform init so identity problems fail fast and clearly.
• Use short, explicit assume-role sessions and refresh them before long jobs to avoid ExpiredToken mid-run.

Quick Command Reference

# Verify identity, bucket existence, and region
aws sts get-caller-identity
aws s3api get-bucket-location --bucket my-tfstate-bucket

# Reproduce Terraform's list call and lock-table access
aws s3api list-objects-v2 --bucket my-tfstate-bucket --prefix env:
aws dynamodb describe-table --table-name my-tf-locks --region us-east-1

# See the raw backend error
TF_LOG=DEBUG terraform init

# Re-initialise / migrate after fixing backend config
terraform init -reconfigure
terraform workspace list

Conclusion

Failed to get existing workspaces is Terraform failing to enumerate remote state before it can do anything else — and the real reason is always in the text after the colon. Read that appended provider error, then check the four usual suspects: a missing s3:ListBucket permission, wrong or nonexistent bucket, region mismatch, and missing or expired credentials (plus DynamoDB lock permissions when a lock table is set). Reproduce the exact list-objects-v2 call with the AWS CLI to isolate the fault, fix the IAM policy or backend config, and re-run terraform init -reconfigure. Codifying the full backend IAM policy keeps this error from reappearing after future refactors.