AWS CLI

AWS failures cluster tightly: the wrong credentials, a policy that denies you, a quota you have hit, or a stack stuck in a state you cannot update. The CLI can tell you which one you are in within about three commands — but only if you know which three. This reference is organized around that, and every section ends at the errors it produces.

Credentials: which identity did you actually use?

Before debugging anything else, prove who you are. Almost every “it works on my laptop but not in CI” report ends here.

The CLI resolves credentials in roughly this order, first match winning:

1. command-line options            --profile, and explicit keys
2. environment variables           AWS_ACCESS_KEY_ID, AWS_PROFILE, AWS_SESSION_TOKEN
3. assumed role / SSO / credential_process   (from ~/.aws/config)
4. shared credentials file         ~/.aws/credentials
5. CLI config file                 ~/.aws/config
6. container credentials           ECS / EKS task role
7. instance metadata (IMDS)        EC2 instance profile

Temporary credentials expire — STS sessions are commonly an hour — which produces ExpiredToken mid-script. Signature failures are usually one of two things: the wrong secret key, or clock skew greater than about five minutes between your machine and AWS, which gives SignatureDoesNotMatch and invalid signature / clock skew. Related: unable to locate credentials, InvalidClientTokenId and AssumeRole denied.

IAM: proving what is actually denied

An AccessDenied is a decision, and the decision has a documented order. Knowing it tells you where to look instead of adding permissions at random.

1. explicit Deny anywhere            → denied, always, no exceptions
2. Service Control Policy (Orgs)     → must ALLOW, or denied
3. permission boundary               → must ALLOW, or denied
4. identity policy (user/role)       → must ALLOW
5. resource policy (bucket, KMS, …)  → can grant across accounts

The practical consequence: adding an Allow never overrides a Deny. If an SCP or a boundary denies the action, no amount of policy on the role will help — and that is exactly why “I gave it AdministratorAccess and it still fails” happens.

Assuming a role has two independent gates that people conflate: the role’s trust policy must allow your principal to assume it, and your identity policy must allow sts:AssumeRole on that role ARN. Failing either gives the same message. See access denied / not authorized, SCP explicit deny, UnauthorizedOperation on EC2 and malformed policy document.

S3: the permission quirks that look like bugs

S3 has two behaviours that reliably send people down the wrong path.

# Does the bucket exist and can I reach it in this region?
aws s3api head-bucket --bucket <b>

# Which region is it actually in? Wrong endpoint = PermanentRedirect
aws s3api get-bucket-location --bucket <b>

# What do the policies say?
aws s3api get-bucket-policy --bucket <b> --query Policy --output text | jq .
aws s3api get-public-access-block --bucket <b>

# List with an explicit prefix rather than scanning the whole bucket
aws s3 ls s3://<b>/path/ --human-readable --summarize

The second quirk is regional: an S3 request sent to the wrong regional endpoint returns PermanentRedirect rather than transparently following. Related: NoSuchBucket, HeadObject forbidden, BucketAlreadyExists — bucket names are globally unique across all of AWS — conflicting conditional operation and SlowDown / 503.

EC2, networking and the “still in use” family

# Why can't I delete this? Find what still references it.
aws ec2 describe-network-interfaces \
  --filters Name=subnet-id,Values=<subnet> \
  --query "NetworkInterfaces[].{id:NetworkInterfaceId, desc:Description, status:Status}"

# Free addresses left in a subnet
aws ec2 describe-subnets --subnet-ids <subnet> \
  --query "Subnets[].{cidr:CidrBlock, free:AvailableIpAddressCount}"

# Security groups referencing this one
aws ec2 describe-security-groups \
  --filters Name=ip-permission.group-id,Values=<sg> \
  --query "SecurityGroups[].GroupId"

DependencyViolation always means something still points at this. The three usual culprits are a network interface you did not create directly (a load balancer, a NAT gateway, a Lambda in a VPC), a security group referenced by another security group’s rules, or a subnet still holding ENIs. See DependencyViolation, ENI insufficient free addresses, EBS volume in use, invalid subnet not found and invalid group not found.

Quotas, capacity and throttling

Three failures that look alike and need different responses — the same trap as every other cloud.

# What are my actual limits?
aws service-quotas list-service-quotas --service-code ec2 \
  --query "Quotas[?contains(QuotaName,'vCPU')].{name:QuotaName, value:Value}" --output table

# Current value for one quota, and whether it is adjustable
aws service-quotas get-service-quota --service-code ec2 --quota-code L-1216C47A

For throttling, let the SDK do the work rather than writing your own sleep loop:

export AWS_RETRY_MODE=adaptive     # or 'standard'
export AWS_MAX_ATTEMPTS=10

adaptive adds client-side rate limiting on top of exponential backoff, which is what you want for a script fanning out across many resources. See vCPU limit exceeded, instance limit exceeded, insufficient instance capacity, throttling / rate exceeded, EC2 address limit and OptInRequired.

CloudFormation: stack states that block you

The genuine cause is always in the stack events, not in the summary status:

aws cloudformation describe-stack-events --stack-name <s> \
  --query "StackEvents[?ResourceStatus=='CREATE_FAILED'].{res:LogicalResourceId, reason:ResourceStatusReason}" \
  --output table

Guides: ROLLBACK_COMPLETE, UPDATE_ROLLBACK_FAILED, insufficient capabilities, no updates to perform and rate exceeded.

Waiters, querying and output

Waiters poll with a bounded number of attempts, so a resource that takes longer than the waiter allows produces ResourceNotReady / max attempts exceeded — which means “still not ready”, not “failed”.

Troubleshooting specific errors

The AWS failures engineers hit most often, each with a dedicated guide:

• S3 NoSuchBucket — wrong name, wrong account, or wrong region endpoint.
• ExpiredToken — temporary credentials aged out mid-run.
• S3 HeadObject forbidden — often a missing key plus a missing s3:ListBucket.
• AccessDenied / not authorized — walk the evaluation order; an explicit Deny always wins.
• AssumeRole denied — trust policy and identity policy are two separate gates.
• SignatureDoesNotMatch and clock skew — wrong secret, or the clock.
• ThrottlingException — back off; set AWS_RETRY_MODE=adaptive.
• DependencyViolation — something still references the resource.
• InsufficientInstanceCapacity — capacity, not quota. Try another AZ.
• CloudFormation ROLLBACK_COMPLETE — delete and recreate; you cannot update out of it.
• Lambda task timed out and too many requests — timeout versus concurrency limit.
• EKS unauthorized and nodes NotReady — IAM-to-Kubernetes mapping, then node health.
• DynamoDB provisioned throughput exceeded — hot partition or under-provisioned table.
• ELB health checks failed — the target is up but failing the check’s path or port.

For anything else, browse the error library or paste the message into the Incident Assistant.

Production checklist

• aws sts get-caller-identity first, every time something is unexpectedly denied.
• aws configure list to find the credential SOURCE. Environment variables outrank profiles.
• Be explicit in scripts — --profile and --region, never ambient defaults.
• Walk the IAM evaluation order before adding permissions. An explicit Deny cannot be out-granted.
• Use --dry-run on EC2 to test permission without side effects.
• Grant s3:ListBucket so a missing object returns 404 instead of a misleading 403.
• --filters before --query. One is server-side; the other downloads everything first.
• AWS_RETRY_MODE=adaptive for anything that fans out across many resources.
• Read describe-stack-events, not the stack status, for the real CloudFormation failure.
• Know that ROLLBACK_COMPLETE means delete and recreate — do not keep retrying the deploy.
• Prefer roles over long-lived access keys, and rotate any key you cannot yet remove.

Frequently asked questions

Why does the CLI use different credentials than I expect?

Because something earlier in the resolution chain won. Command-line options beat environment variables, which beat your profile, which beats the instance role — so a stale AWS_ACCESS_KEY_ID or AWS_PROFILE exported in your shell will silently override --profile prod. Run aws configure list: unlike get-caller-identity, it prints the source of each value, so you can see immediately whether the region came from the environment, the config file, or IMDS.

I have AdministratorAccess and still get AccessDenied. Why?

Almost certainly an explicit Deny above your identity policy. IAM evaluates in a fixed order and an explicit Deny anywhere — in a Service Control Policy at the organization level, in a permission boundary, or in a resource policy — wins over any Allow, including administrator. Check SCPs first if the account is in an Organization, then permission boundaries on the role. Adding more permissions cannot fix a Deny.

Why does S3 return 403 for an object that does not exist?

Because you lack s3:ListBucket. Without it, S3 will not distinguish “absent” from “not yours” — confirming a 404 would leak whether the key exists to someone who cannot enumerate the bucket. The practical effect is that a simple typo in the key looks exactly like a permissions failure. Grant s3:ListBucket on the bucket ARN (not the object ARN) and the same request starts returning an honest NoSuchKey.

What is the difference between a quota error and a capacity error?

A quota error — VcpuLimitExceeded, InstanceLimitExceeded — means your account is not permitted that many, and the fix is a limit increase through Service Quotas. InsufficientInstanceCapacity means AWS itself has none of that instance type free in that Availability Zone right now; no limit increase helps, and the fix is another AZ, another instance family, or another region. ThrottlingException is a third thing again: you are calling the API too fast, and the answer is backoff, not more concurrency.

How do I get out of ROLLBACK_COMPLETE?

Delete the stack and create it again — CloudFormation does not allow updating out of that state. It only occurs when the very first create failed, so there is nothing of value to preserve. Do not confuse it with UPDATE_ROLLBACK_FAILED, which happens on a later update and is recoverable with continue-update-rollback (optionally skipping the resources that will not roll back). In both cases the useful detail is in describe-stack-events, not the status.

Why is my script being throttled when it only reads?

Reads count against API rate limits too, and the usual cause is client-side filtering. --query is applied after the response arrives, so a loop of describe-* calls filtered with --query still requests everything, every time. Move the filtering server-side with --filters, request only the fields you need, and set AWS_RETRY_MODE=adaptive with AWS_MAX_ATTEMPTS so the SDK backs off properly instead of hammering.

Related resources

• Guide: Azure CLI — the same operational ground on Azure; the identity and quota patterns rhyme closely.
• Guide: Cloud Security — least privilege and the identity model underneath the IAM section here.
• Guide: Kubernetes Security — where EKS workloads pick up from cluster-side controls.
• Tool: Incident Assistant — paste an AWS error and get an ordered triage plan.