🚨 AWS Error Guide: 'ResourceInitializationError: unable to pull secrets' — ECS Task Startup Failure

Overview

Before an ECS (especially Fargate) task’s containers start, the agent must fetch things on the task’s behalf: image registry auth, Secrets Manager/SSM secret values referenced in the task definition, and environment files from S3. If any of those fetches fail — usually a missing permission on the task execution role or no network path to the service — the task stops during provisioning with ResourceInitializationError: unable to pull secrets or registry auth. The container never runs, so there are no application logs to look at.

You will see it as the task’s stoppedReason:

ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secret from asm: service call has been retried 5 time(s): failed to fetch secret arn:aws:secretsmanager:us-east-1:REDACTED:secret:db-pass ...

The registry-auth and env-file variants read:

ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed ... dial tcp: i/o timeout

It occurs on task start when the execution role lacks a permission, the referenced secret/KMS key is inaccessible, or the task’s subnet has no route to Secrets Manager/SSM/ECR/S3.

Symptoms

• Tasks flip PENDING → STOPPED immediately with stoppedReason containing ResourceInitializationError and unable to pull secrets or registry auth.
• No application/container logs appear, because containers never started.
• Service events show repeated task launches failing the same way, and the service never reaches steady state.
• Removing the secrets/repositoryCredentials block lets the task start, confirming the failure is in the fetch step.

aws ecs describe-tasks --cluster prod --tasks <task-arn> \
  --query 'tasks[0].stoppedReason' --output text

ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secret from asm ... AccessDeniedException

Common Root Causes

1. Execution role missing secret/SSM read permission

The task execution role (not the task role) needs secretsmanager:GetSecretValue or ssm:GetParameters for the referenced ARNs.

ROLE=$(aws ecs describe-task-definition --task-definition app:7 \
  --query 'taskDefinition.executionRoleArn' --output text | awk -F/ '{print $NF}')
aws iam simulate-principal-policy --policy-source-arn arn:aws:iam::REDACTED:role/$ROLE \
  --action-names secretsmanager:GetSecretValue \
  --resource-arns arn:aws:secretsmanager:us-east-1:REDACTED:secret:db-pass \
  --query 'EvaluationResults[0].EvalDecision' --output text

implicitDeny

The execution role cannot read the secret — the fetch fails with AccessDenied.

2. Missing KMS decrypt permission on the secret’s key

If the secret is encrypted with a customer-managed KMS key, the execution role also needs kms:Decrypt on that key.

aws secretsmanager describe-secret --secret-id db-pass --query 'KmsKeyId' --output text

arn:aws:kms:us-east-1:REDACTED:key/REDACTED

A CMK is in use, so the role needs decrypt on it in addition to GetSecretValue.

3. No network path to the service (private subnet, no endpoint/NAT)

A Fargate task in a private subnet with no NAT gateway and no VPC endpoint for Secrets Manager/SSM/ECR/S3 cannot reach the service, giving i/o timeout.

aws ec2 describe-vpc-endpoints \
  --filters Name=vpc-id,Values=vpc-REDACTED \
  --query "VpcEndpoints[].ServiceName" --output text

com.amazonaws.us-east-1.ecr.dkr	com.amazonaws.us-east-1.ecr.api

ECR endpoints exist but there is no secretsmanager endpoint and no NAT — the secret fetch times out.

4. Missing S3 access for image layers or env files

ECR image layers and env files live in S3; without the S3 Gateway endpoint (or NAT) and s3:GetObject, the pull fails.

aws ec2 describe-vpc-endpoints \
  --filters Name=vpc-id,Values=vpc-REDACTED Name=service-name,Values=com.amazonaws.us-east-1.s3 \
  --query 'length(VpcEndpoints)' --output text

0

No S3 Gateway endpoint, so layer/env-file retrieval has no private path.

5. Wrong secret ARN or region in the task definition

A typo, a wrong-region ARN, or referencing a secret that was rotated/deleted makes the fetch fail even with correct permissions.

aws ecs describe-task-definition --task-definition app:7 \
  --query 'taskDefinition.containerDefinitions[].secrets[].valueFrom' --output text

arn:aws:secretsmanager:us-west-2:REDACTED:secret:db-pass

The task runs in us-east-1 but the secret ARN points at us-west-2 — no such secret in-region.

Diagnostic Workflow

Step 1: Read the exact stoppedReason

aws ecs describe-tasks --cluster <cluster> --tasks <task-arn> \
  --query 'tasks[0].[stoppedReason,containers[0].reason]' --output text

The wording tells you which fetch failed — retrieve secret from asm (Secrets Manager), ssm (Parameter Store), or ecr registry auth (image pull).

Step 2: Identify the execution role and simulate the needed action

aws ecs describe-task-definition --task-definition <family:rev> \
  --query 'taskDefinition.executionRoleArn' --output text
aws iam simulate-principal-policy --policy-source-arn <exec-role-arn> \
  --action-names secretsmanager:GetSecretValue kms:Decrypt \
  --resource-arns <secret-arn> <kms-key-arn> \
  --query 'EvaluationResults[].[EvalActionName,EvalDecision]' --output text

Any implicitDeny/explicitDeny is a permission gap on the execution role.

Step 3: Distinguish AccessDenied from a network timeout

An AccessDeniedException in the reason is IAM/KMS; an i/o timeout / dial tcp is a networking (endpoint/NAT/security-group) problem.

aws ecs describe-tasks --cluster <cluster> --tasks <task-arn> \
  --query 'tasks[0].stoppedReason' --output text | grep -oE 'AccessDenied|i/o timeout|dial tcp'

Step 4: Verify the network path for private-subnet tasks

aws ec2 describe-vpc-endpoints --filters Name=vpc-id,Values=<vpc-id> \
  --query 'VpcEndpoints[].ServiceName' --output text
aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=<task-subnet> \
  --query 'RouteTables[].Routes[?NatGatewayId!=null].NatGatewayId' --output text

For a private task you need either interface endpoints for secretsmanager/ssm/ecr.api/ecr.dkr + an S3 Gateway endpoint, or a NAT route.

Example Root Cause Analysis

A Fargate service in private subnets rolled out a new task definition that injected a DB password from Secrets Manager. Every task stopped with ResourceInitializationError ... unable to retrieve secret from asm ... i/o timeout.

The reason said i/o timeout, not AccessDenied, pointing at networking rather than IAM:

aws ecs describe-tasks --cluster prod --tasks <task-arn> \
  --query 'tasks[0].stoppedReason' --output text | grep -oE 'i/o timeout|AccessDenied'

i/o timeout

The VPC had ECR and S3 endpoints (so images pulled fine before) but no Secrets Manager interface endpoint, and the private subnets had no NAT:

aws ec2 describe-vpc-endpoints --filters Name=vpc-id,Values=vpc-REDACTED \
  --query 'VpcEndpoints[].ServiceName' --output text | tr '\t' '\n' | grep secretsmanager

No secretsmanager endpoint returned. Adding an interface endpoint for com.amazonaws.us-east-1.secretsmanager (with a security group allowing 443 from the task ENIs and Private DNS enabled) gave the tasks a private path, and they started cleanly. The image had worked all along because ECR/S3 endpoints already existed — only the new secret fetch lacked a route.

Prevention Best Practices

• Grant the task execution role exactly the secret/SSM ARNs it reads plus kms:Decrypt on the secret’s CMK — this is separate from the task role your app uses.
• For private-subnet Fargate, provision interface endpoints for secretsmanager, ssm, ecr.api, ecr.dkr, and logs, plus an S3 Gateway endpoint (with Private DNS on), or a NAT route.
• Open the endpoint security groups to 443 from the task ENIs’ security group so the fetch can connect.
• Validate secret/env-file ARNs and regions in the task definition at deploy time; a wrong-region ARN fails even with correct IAM.
• Test a task-definition change in a canary service first so a fetch failure does not stop the whole service from reaching steady state.

Quick Command Reference

# Why did the task stop?
aws ecs describe-tasks --cluster <cluster> --tasks <task-arn> \
  --query 'tasks[0].[stoppedReason,containers[0].reason]' --output text

# Execution role and its secret/KMS permissions
aws ecs describe-task-definition --task-definition <family:rev> \
  --query 'taskDefinition.executionRoleArn' --output text
aws iam simulate-principal-policy --policy-source-arn <exec-role-arn> \
  --action-names secretsmanager:GetSecretValue kms:Decrypt \
  --resource-arns <secret-arn> <kms-key-arn>

# AccessDenied vs network timeout
aws ecs describe-tasks --cluster <cluster> --tasks <task-arn> \
  --query 'tasks[0].stoppedReason' --output text | grep -oE 'AccessDenied|i/o timeout'

# VPC endpoints present for the task's VPC
aws ec2 describe-vpc-endpoints --filters Name=vpc-id,Values=<vpc-id> \
  --query 'VpcEndpoints[].ServiceName' --output text

Conclusion

ResourceInitializationError: unable to pull secrets or registry auth means ECS could not fetch a secret, registry credential, or env file before the container started. The usual root causes:

1. The execution role missing secretsmanager:GetSecretValue/ssm:GetParameters for the ARN.
2. Missing kms:Decrypt on the secret’s customer-managed key.
3. No network path (no NAT and no interface endpoint) from a private subnet.
4. Missing S3 Gateway endpoint/access for image layers or env files.
5. A wrong-region or stale secret ARN in the task definition.

Read the stoppedReason to tell AccessDenied from an i/o timeout, then fix the execution-role permissions (including KMS) or provision the interface/Gateway endpoints the task needs — the container only runs once every pre-start fetch succeeds.