🚨 Loki Error Guide: 'NoCredentialProviders: no valid providers in chain' — Give Loki Working Object-Store Credentials

Overview

Loki logs this error when the AWS SDK cannot resolve any credentials from its provider chain before talking to the object store:

NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

It commonly surfaces wrapped as a CredentialsError on flush:

level=error msg="failed to flush" err="CredentialsError: failed to refresh cached credentials, no valid providers in chain"

Ingesters emit it on chunk flush, the compactor on compaction, and queriers on reads. The SDK walks a chain — static keys in config, environment variables, the web-identity token (IRSA/Workload Identity), then the instance/metadata profile — and this error means every link came up empty. It is not a 403: the request never authenticated at all because no credential was found to sign it.

Symptoms

• Ingesters log NoCredentialProviders ... no valid providers in chain or CredentialsError on every flush; loki_ingester_memory_chunks climbs.
• The pod has no AWS_ROLE_ARN or AWS_WEB_IDENTITY_TOKEN_FILE in its environment despite an IRSA/Workload Identity setup.
• The error starts right after removing static keys from config in favour of IRSA that was never actually bound.
• Compactor and querier fail identically because they share the same broken identity.
• An instance-profile-based node worked, but the pod does not inherit it, so only pods fail.

Common Root Causes

• IRSA/Workload Identity not bound to the pod — the ServiceAccount lacks the role annotation, or the pod uses the wrong ServiceAccount, so no web-identity token is projected.
• No static keys and no env credentials — access_key_id/secret_access_key are absent from config and no AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY are set.
• Expired or missing instance profile — the node’s metadata credentials are unavailable (IMDS disabled, hop limit, or no profile attached), so the metadata link of the chain fails.
• Wrong AWS_WEB_IDENTITY_TOKEN_FILE — the token path points at a file that does not exist or is not mounted, so the web-identity provider is skipped.
• Provider chain order — a partially configured provider short-circuits or is not reached, leaving the SDK with nothing usable.

How to diagnose

1. Pull the failing operation and confirm it is a credentials failure, not a 403 or 404:

   kubectl logs -l app=loki,component=ingester --since=10m \
     | grep -iE 'NoCredentialProviders|CredentialsError|no valid providers'

2. Inspect the pod environment for the IRSA/Workload Identity variables the SDK needs:

   kubectl exec -it deploy/loki-ingester -- env \
     | grep -iE 'AWS_ROLE_ARN|AWS_WEB_IDENTITY_TOKEN_FILE|AWS_ACCESS_KEY_ID'

3. Verify the ServiceAccount is annotated with the role and that the pod uses it:

   kubectl get sa loki -o yaml | grep -i 'eks.amazonaws.com/role-arn'
   kubectl get deploy/loki-ingester -o jsonpath='{.spec.template.spec.serviceAccountName}'

4. Confirm the web-identity token is actually mounted at the advertised path:

   kubectl exec -it deploy/loki-ingester -- sh -c \
     'ls -l "$AWS_WEB_IDENTITY_TOKEN_FILE"'

5. Reproduce the credential resolution from inside the pod using the same identity:

   kubectl exec -it deploy/loki-ingester -- aws sts get-caller-identity
   # NoCredentialProviders here confirms the chain resolves nothing

Fixes

Prefer IRSA/Workload Identity and bind it correctly so the SDK’s web-identity provider resolves a token. Annotate the ServiceAccount and point the deployment at it, then confirm AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE appear in the pod:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: loki
  namespace: logging
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/loki-s3

Set static keys from a Secret when IRSA is not available so the config-file link of the chain resolves. Reference the values through env from a Kubernetes Secret rather than hardcoding them:

storage_config:
  aws:
    s3: s3://us-east-1/loki-chunks-prod
    bucketnames: loki-chunks-prod
    region: us-east-1
    access_key_id: ${AWS_ACCESS_KEY_ID}
    secret_access_key: ${AWS_SECRET_ACCESS_KEY}

Inject those keys from a Secret so they land in the pod environment and the SDK can pick them up before falling through the rest of the chain:

env:
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: loki-s3-creds
        key: access_key_id
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: loki-s3-creds
        key: secret_access_key

Fix the credential provider chain order and token file when the web-identity link is being skipped — make sure AWS_WEB_IDENTITY_TOKEN_FILE points at a mounted, readable token and that no half-configured static provider precedes it. Verify end to end from the pod:

kubectl exec -it deploy/loki-ingester -- sh -c \
  'aws sts get-caller-identity && aws s3 ls s3://loki-chunks-prod/'

What to watch out for

• NoCredentialProviders is an authentication absence, not a denial. Widening IAM actions will not help until a credential resolves at all.
• Mixing static keys and IRSA can be confusing: static env credentials take precedence over the web-identity token, so leftover keys can mask or override the role you intended.
• IRSA requires both the ServiceAccount annotation and the pod referencing that ServiceAccount; one without the other silently yields no token.
• If you rely on the instance/metadata profile, IMDS must be reachable from the pod (hop limit, IMDSv2), or the metadata link of the chain fails.
• Rotated Secrets need a pod restart unless mounted for live reload; stale env keys can keep failing after the Secret is fixed.

Related

• Loki Error Guide: ‘AccessDenied: Access Denied status code: 403’ — the next hurdle once credentials resolve but lack the right IAM actions.
• Loki Error Guide: ‘failed to flush chunks’ — the flush-path symptom that a credentials failure produces.
• Loki Error Guide: ‘context deadline exceeded’ — object-store timeouts that can follow credential refresh stalls.