🚨 Loki Error Guide: 'NoSuchBucket: The specified bucket does not exist' — Point Loki at a Bucket That Exists

Overview

Loki logs this error when it asks the object store for a bucket that the store cannot find, and the request is rejected with a 404:

NoSuchBucket: The specified bucket does not exist
	status code: 404, request id=..., host id=...

The ingester emits it on chunk flush, the compactor on retention/compaction, and the querier on reads. Unlike a 403 (AccessDenied), a 404 means the credentials were accepted but the named bucket does not resolve at the endpoint Loki is talking to. That happens because of a typo in bucketnames, a bucket that was never created, a wrong region/endpoint that routes the request to a different account, or a path-style versus virtual-host mismatch on MinIO/Ceph. Chunks stay in memory until the target bucket is real and reachable.

Symptoms

• Ingesters log NoSuchBucket ... status code: 404 on every flush and loki_ingester_memory_chunks climbs as unflushed chunks accumulate.
• The error appears immediately after a fresh deploy into a new environment, before anyone created the bucket.
• Compactor logs failed to compact with a 404 and retention stalls.
• Queries for older data fail while recent in-memory data still returns.
• aws s3 ls s3://<bucket>/ from the pod returns NoSuchBucket for the exact name in bucketnames.

Common Root Causes

• Typo in storage_config.aws.bucketnames — a misspelled or stale bucket name that does not exist in the account.
• Bucket never created — the deployment shipped before the bucket was provisioned in the target environment.
• Wrong region or endpoint — the request is routed to a different region or account where the bucket does not exist, surfacing as a 404.
• Path-style vs virtual-host mismatch — on MinIO/Ceph the virtual-host form bucket.host fails to resolve, so the store reports the bucket as missing.
• Environment drift — bucketnames still points at the staging bucket after promotion to prod, and that bucket is absent in the prod account.

How to diagnose

1. Read the exact bucket name Loki used — pull the failing operation and the name from the logs so you compare against the real bucket, not what you assume the config says:

   kubectl logs -l app=loki,component=ingester --since=10m \
     | grep -iE 'NoSuchBucket|status code: 404|failed to flush'

2. Confirm the bucket actually exists in the target account/region with the AWS CLI:

   aws s3api head-bucket --bucket loki-chunks-prod --region us-east-1
   # exit 0 = exists and reachable; 404 = NoSuchBucket

3. Reproduce from inside the pod using the same identity Loki runs as, so you test the real network path and endpoint:

   kubectl exec -it deploy/loki-ingester -- sh -c \
     'aws s3 ls s3://loki-chunks-prod/'

4. Inspect the storage config for bucketnames, region, endpoint, and s3forcepathstyle:

   storage_config:
     aws:
       s3: s3://us-east-1/loki-chunks-prod
       bucketnames: loki-chunks-prod
       region: us-east-1
       endpoint: s3.us-east-1.amazonaws.com
       s3forcepathstyle: false

5. Probe the endpoint directly to see whether the host resolves the bucket path-style or virtual-host:

   curl -sI https://s3.us-east-1.amazonaws.com/loki-chunks-prod/
   # 404 with a NoSuchBucket XML body confirms the store cannot find it

Fixes

Create the bucket or correct the name so bucketnames matches a bucket that actually exists in the target account and region. If the bucket was never provisioned, create it, then let Loki retry its queued flushes:

aws s3api create-bucket \
  --bucket loki-chunks-prod \
  --region us-east-1 \
  --create-bucket-configuration LocationConstraint=us-east-1

Set the correct region and endpoint so the request is not routed to an account where the bucket is absent. Align region and endpoint with where the bucket really lives:

storage_config:
  aws:
    s3: s3://us-east-1/loki-chunks-prod
    bucketnames: loki-chunks-prod
    region: us-east-1
    endpoint: s3.us-east-1.amazonaws.com
    s3forcepathstyle: false

Force path-style addressing on MinIO/Ceph so the bucket is addressed as endpoint/bucket rather than bucket.endpoint, which self-hosted stores usually cannot resolve:

storage_config:
  aws:
    s3: s3://tenant-a:secret@minio.storage.svc:9000/loki-chunks-prod
    bucketnames: loki-chunks-prod
    endpoint: minio.storage.svc:9000
    region: us-east-1
    s3forcepathstyle: true
    insecure: true

Verify from the pod before declaring it fixed — run the same list/write/delete cycle against the corrected bucket using Loki’s identity, so a config typo or endpoint mistake surfaces immediately:

kubectl exec -it deploy/loki-ingester -- sh -c \
  'aws s3 ls s3://loki-chunks-prod/ && \
   echo probe | aws s3 cp - s3://loki-chunks-prod/_probe/p.txt && \
   aws s3 rm s3://loki-chunks-prod/_probe/p.txt'

What to watch out for

• A 404 is not a permissions problem. Do not add IAM actions to fix NoSuchBucket; the credentials were accepted and the bucket simply does not resolve.
• A wrong region or endpoint can masquerade as a missing bucket because the request lands in the wrong account; verify the bucket exists in the region Loki targets, not just that it exists somewhere.
• On MinIO/Ceph, leaving s3forcepathstyle: false is the most common cause; self-hosted stores almost always need true.
• After promotion between environments, confirm bucketnames was updated; a stale staging name is a frequent hidden source of prod 404s.
• Unflushed chunks live in ingester memory while the bucket is unreachable, so a lingering 404 can OOM ingesters; alert on rising loki_ingester_memory_chunks.

Related

• Loki Error Guide: ‘AccessDenied: Access Denied status code: 403’ — the permissions counterpart; a 403 means the bucket exists but the identity is refused.
• Loki Error Guide: ‘failed to flush chunks’ — the upstream symptom when a bucket problem stalls the flush path.
• Loki Error Guide: ‘context deadline exceeded’ — object-store timeouts that can follow a misrouted endpoint.