🚨 Loki Error Guide: 'chunk not found' — Reconcile the Index with What Actually Lives in Object Storage

Overview

Loki’s querier logs this when the index points at a chunk object that no longer exists — or was never written — in object storage:

level=error msg="error fetching chunks" err="chunk not found"

The same underlying condition surfaces with slightly different wording depending on the store, for example entry not found in index or an object-store object doesn't exist / NoSuchKey wrapped by the fetch. The index is a map from stream and time range to a set of chunk keys; the querier reads that map, then fetches each key from the chunk store. When the map and the store disagree — the index still lists a chunk that retention deleted, that landed in a different bucket, or that failed to flush — the fetch fails and the query errors or returns partial results. The fix is always to reconcile the two: make the index and the chunk store agree on what exists and where.

Symptoms

• Queries over a specific time window fail with chunk not found while adjacent windows return normally.
• The error names a chunk key; listing that exact key in the bucket returns NoSuchKey / object doesn't exist.
• Failures cluster right after a retention or compaction run, or right after a storage config change.
• Recently ingested data queries fine, but older data intermittently errors — pointing at retention or a failed historical flush.
• Restarting queriers does not help, because the missing object is genuinely absent from storage, not cached away.

Common Root Causes

• Retention or the compactor deleted chunks the index still references — a rogue or duplicate compactor applied retention while the index entries were not cleaned up consistently, leaving dangling references.
• Index and chunk storage_config point at different buckets or prefixes — the querier reads index entries that name chunk keys under a bucket/prefix the chunk store is not actually using.
• Stale index or results cache — a cached index page still lists chunks that have since been deleted, so the querier tries to fetch objects that are already gone.
• A chunk that failed to flush — the index was written but the corresponding object never landed in storage because the flush errored.
• Manual or external object deletion — a lifecycle rule, a cleanup script, or a human removed objects out from under Loki.

How to diagnose

1. Read the error and extract the missing chunk key, then confirm whether the object actually exists in the bucket the chunk store uses:

   kubectl logs -l app=loki,component=querier --tail=200 \
     | grep -i 'chunk not found\|object doesn'\''t exist'

2. List the exact object the index referenced, using the same credentials/identity the querier runs with:

   # Substitute the key from the log line
   aws s3 ls s3://my-loki-bucket/fake/01H8Z.../
   aws s3api head-object --bucket my-loki-bucket --key chunks/01H8Z... || echo "object is genuinely missing"

3. Compare the index and chunk store configuration — confirm both resolve to the same bucket and prefix, and that no recent change split them apart:

   curl -s http://loki:3100/config | grep -A25 'storage_config'

4. Check compactor / retention activity around the failure timestamps to see whether a deletion run lines up with the missing chunks:

   kubectl logs -l app=loki,component=compactor --tail=300 \
     | grep -iE 'retention|delete|mark for deletion'

5. Confirm the storage config resolves index and chunks consistently — a single bucket/prefix pair, one active compactor:

   storage_config:
     aws:
       s3: s3://us-east-1/my-loki-bucket
     tsdb_shipper:
       active_index_directory: /loki/tsdb-index
       cache_location: /loki/tsdb-cache

Fixes

Confirm whether the object exists before anything else. If aws s3 ls shows the key is genuinely gone, this is a data-consistency problem, not a query bug — do not tune query settings. If the object does exist, the problem is a bucket/prefix or cache mismatch, and the fix is configuration:

aws s3 ls s3://my-loki-bucket/ --recursive | grep 01H8Z
# present -> config/cache mismatch; absent -> retention/flush data loss

Align the index and chunk storage_config to the same bucket and prefix. A split between where index entries are written and where chunks live is the classic cause of dangling references — pin both to one location:

storage_config:
  aws:
    s3: s3://us-east-1/my-loki-bucket
    bucketnames: my-loki-bucket
  tsdb_shipper:
    shared_store: s3
schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

Ensure exactly one compactor manages retention. Two compactors, or retention enabled in multiple places, delete chunks inconsistently and leave the index referencing gone objects. Run a single instance:

compactor:
  working_directory: /loki/compactor
  shared_store: s3
  retention_enabled: true
  compaction_interval: 10m
  # Run this as a single replica only

Flush and clear stale caches so the querier stops reading a cached index that lists deleted chunks. Restart the query path or flush the index/results cache, then re-run the query:

# Redis-backed results/index cache: flush the affected keys
redis-cli -h loki-results-cache FLUSHDB
kubectl rollout restart deploy/loki-querier

Check for failed flushes as the upstream cause. If chunks are missing because a flush errored, the index entry exists with no object behind it — fix the flush path first, then the missing-chunk errors stop appearing for new data:

kubectl logs -l app=loki,component=ingester --tail=300 | grep -i 'failed to flush'

What to watch out for

• chunk not found where the object is genuinely absent means data loss — no config change brings it back; focus on stopping the bleeding (retention, flushes) so it does not recur.
• Never run more than one compactor or enable retention in multiple components; inconsistent deletion is the most common cause of dangling index references.
• A bucket lifecycle policy silently expiring objects will masquerade as a Loki bug — audit S3 lifecycle rules before blaming the index.
• Splitting index and chunk storage across buckets or prefixes during a migration will produce this error until both sides agree; migrate atomically or keep both pointed at one location.
• Stale caches produce transient chunk not found that clears on its own; if the error persists after a cache flush, the object is truly missing.

Related

• Loki Error Guide: ‘failed to flush chunks’ — a failed flush leaves an index entry with no object behind it, the upstream cause of missing chunks.
• Loki Error Guide: ‘access denied’ to the object store — a permissions gap can look like a missing object when the querier cannot read it.
• Loki Error Guide: ‘too many chunks to fetch’ — the opposite fetch-path problem, useful when tuning how the querier reads chunks from storage.