🚨 Loki Error Guide: 'AccessDenied: Access Denied status code: 403' — Fix Object Storage Permissions

Overview

Loki cannot read or write objects because the object store rejected the request with a 403. Ingesters, compactor, and queriers log:

level=error msg="failed to flush" err="AccessDenied: Access Denied\n\tstatus code: 403, request id=..., host id=..."

The equivalent on GCS/Azure reads storage: object doesn't exist / permission denied or 403 Forbidden. Object storage is Loki’s system of record for chunks and (for shipper/TSDB) the index, so a persistent 403 stalls flushes, blocks compaction, and eventually fails queries.

Symptoms

• Ingesters log failed to flush chunks ... AccessDenied ... 403; unflushed chunks pile up in memory.
• loki_ingester_wal_disk_full_failures_total or growing loki_ingester_memory_chunks as flushes back up.
• Compactor logs failed to compact / 403 and retention/compaction stalls.
• Queries for older data fail while recent in-memory data still returns.
• The error starts after a credential rotation, bucket/policy change, or new deployment in a new environment.

Common Root Causes

• IAM policy missing actions — the role lacks s3:PutObject, s3:GetObject, s3:DeleteObject, or s3:ListBucket on the bucket/prefix.
• Wrong bucket name or prefix — typo in bucketnames, or the policy scoped to a different prefix than Loki writes.
• Region mismatch — region/endpoint pointing to the wrong region returns 403 on some providers.
• Expired or wrong credentials — a rotated key still referenced, or IRSA/Workload Identity not bound to the pod.
• Bucket policy or SCP deny — an explicit deny (encryption, VPC endpoint, or org SCP) overriding the role.
• KMS permissions missing — the object is SSE-KMS encrypted but the role lacks kms:Decrypt/kms:GenerateDataKey.

Diagnostic Workflow

Confirm the exact operation and object from Loki’s logs:

kubectl logs -l app.kubernetes.io/component=ingester --since=10m \
  | grep -iE 'AccessDenied|403|failed to flush'

Check Loki’s storage config for bucket, region, and auth:

storage_config:
  aws:
    s3: s3://us-east-1/loki-chunks-prod
    bucketnames: loki-chunks-prod
    region: us-east-1
    s3forcepathstyle: false

Test the same permissions from inside the pod (or an equivalent identity):

kubectl exec -it deploy/loki-ingester -- sh -c \
  'aws s3 ls s3://loki-chunks-prod/ && \
   echo hi | aws s3 cp - s3://loki-chunks-prod/_perm_test/probe.txt && \
   aws s3 rm s3://loki-chunks-prod/_perm_test/probe.txt'

Verify the IRSA/Workload Identity binding is actually attached:

kubectl exec -it deploy/loki-ingester -- env | grep -iE 'AWS_ROLE_ARN|AWS_WEB_IDENTITY'

The minimum S3 policy Loki needs (chunks + index bucket/prefix):

{
  "Effect": "Allow",
  "Action": ["s3:PutObject","s3:GetObject","s3:DeleteObject","s3:ListBucket"],
  "Resource": [
    "arn:aws:s3:::loki-chunks-prod",
    "arn:aws:s3:::loki-chunks-prod/*"
  ]
}

If SSE-KMS is enabled, also grant kms:Decrypt and kms:GenerateDataKey on the key.

Example Root Cause Analysis

A newly promoted prod Loki started logging AccessDenied ... 403 on every flush, and loki_ingester_memory_chunks climbed steadily as unflushed data accumulated. Running the in-pod probe, aws s3 ls succeeded but aws s3 cp failed with 403 — so the identity could list and read but not write. The IRSA role’s policy granted s3:GetObject/s3:ListBucket but omitted s3:PutObject and s3:DeleteObject, which had been fine in staging where an over-broad admin role was used.

Adding s3:PutObject and s3:DeleteObject (the latter needed for compaction/retention) on both arn:aws:s3:::loki-chunks-prod and .../* cleared the 403 immediately; queued chunks flushed within minutes and memory_chunks dropped back to baseline. The team also discovered the bucket used SSE-KMS, so they added kms:GenerateDataKey/kms:Decrypt to prevent a follow-on 403 once the object-level permissions were correct.

Prevention Best Practices

• Grant exactly the four S3 actions Loki needs (PutObject, GetObject, DeleteObject, ListBucket) on both the bucket ARN and the /* object ARN.
• If using SSE-KMS, include kms:Decrypt and kms:GenerateDataKey on the key.
• Use IRSA/Workload Identity rather than long-lived keys, and verify the binding is present in the pod env.
• Match region/endpoint/bucketnames exactly to the real bucket; test with an in-pod probe before go-live.
• Alert on flush failures (failed to flush) and rising loki_ingester_memory_chunks so a 403 doesn’t silently fill memory.
• Review bucket policies and org SCPs for explicit denies (VPC endpoint, encryption enforcement).

Quick Command Reference

# Loki's own 403/flush errors
kubectl logs -l app.kubernetes.io/component=ingester --since=10m | grep -iE 'AccessDenied|403'

# Probe read+write+delete from the pod's identity
kubectl exec -it deploy/loki-ingester -- sh -c \
  'echo t | aws s3 cp - s3://loki-chunks-prod/_perm_test/p && aws s3 rm s3://loki-chunks-prod/_perm_test/p'

# Confirm IRSA binding
kubectl exec -it deploy/loki-ingester -- env | grep AWS_ROLE_ARN

Conclusion

AccessDenied: Access Denied status code: 403 means the object store refused Loki’s request — almost always an IAM/policy gap, a bucket/region/prefix mismatch, or missing KMS permissions. Read Loki’s logs for the failed operation, reproduce it with an in-pod probe to isolate read vs write vs delete, and grant exactly the S3 (and KMS) actions Loki needs on the correct bucket ARNs. Because object storage is Loki’s system of record, alert on flush failures and rising in-memory chunks so a permissions error never silently backs up ingestion.