🚨 Loki Error Guide: 'invalid magic number' — Find and Remove the Corrupt Chunk or Index File

Overview

Loki tags its on-disk chunk and index formats with a fixed magic-number header. When it reads a file whose first bytes do not match that header, decoding aborts with:

level=error msg="error fetching chunks" err="invalid magic number 0"

It also appears on the index side when opening a boltdb/TSDB file, for example:

error creating index client: opening boltdb file: invalid magic number

A magic-number mismatch means the bytes are not what Loki expects at all — the object is truncated, corrupt, or is simply the wrong file at that path. This is not a tuning problem: some specific object or local file is bad. The work is to identify which one, remove or replace it, and let Loki recover from a healthy replica or re-download a clean copy. The 0 in invalid magic number 0 is a strong tell that the file was empty or truncated before the header was ever written.

Symptoms

• A subset of queries fail with invalid magic number while most of the store reads fine.
• An ingester or querier fails to open a local boltdb/TSDB index file at startup and crash-loops.
• The error references a specific chunk key or index file path — the corruption is localized, not store-wide.
• The failure began right after an interrupted upload, a node crash, an OOM kill, or a disk incident.
• Deleting the referenced object or wiping the local index cache lets the component start and query cleanly again.

Common Root Causes

• Partial or interrupted upload — a flush or shipper upload was cut off, leaving a truncated object in the bucket with no valid header.
• Wrong file at the path — a misconfigured prefix, a migration, or an external tool placed a non-Loki file where a chunk or index is expected.
• Disk corruption on a local boltdb/TSDB file — a failing volume or an unclean write corrupted the ingester/querier’s local index copy.
• Version or format mismatch — the data was written with one schema/store (for example boltdb-shipper) but is being read as another (TSDB), so the header does not match.
• Empty object — a zero-byte file (often the invalid magic number 0 case) from a failed write that created the key but wrote no content.

How to diagnose

1. Pull the failing object key or index file path out of the log line so you know exactly what is corrupt:

   kubectl logs -l app=loki,component=querier --tail=200 \
     | grep -i 'invalid magic number'

2. Check the object’s size and header in storage — a zero-byte or tiny object is almost certainly truncated:

   aws s3api head-object --bucket my-loki-bucket --key chunks/01H8Z...
   # ContentLength near 0 confirms a truncated/empty object
   aws s3 cp s3://my-loki-bucket/chunks/01H8Z... - | head -c 16 | xxd

3. Inspect the local index directory on the affected pod if the error is on a boltdb/TSDB file:

   kubectl exec -it loki-ingester-0 -- ls -la /loki/tsdb-index /loki/boltdb-shipper-active
   kubectl exec -it loki-ingester-0 -- du -sh /loki/tsdb-cache

4. Confirm the schema matches the data — a store/object_store mismatch reads good bytes as the wrong format:

   curl -s http://loki:3100/config | grep -A15 'schema_config'

5. Verify the corruption is localized, not a store-wide problem, by querying a different time range that uses different objects:

   schema_config:
     configs:
       - from: 2024-01-01
         store: tsdb
         object_store: s3
         schema: v13

Fixes

Identify and remove the corrupt object. Once head-object confirms a truncated or zero-byte chunk, delete it so Loki stops trying to decode it. With a replication factor greater than one, another ingester still holds the data and it can be re-flushed or served from a peer:

aws s3api head-object --bucket my-loki-bucket --key chunks/01H8Z...   # confirm it is bad
aws s3 rm s3://my-loki-bucket/chunks/01H8Z...                          # remove the corrupt object

Rely on replication for recovery. If replication_factor is 3, the chunk existed on multiple ingesters and the loss of one copy is not fatal — verify replication so a single corrupt object is never a single point of failure:

common:
  replication_factor: 3
ingester:
  lifecycler:
    ring:
      replication_factor: 3

Ensure atomic writes to storage. Interrupted uploads are the leading source of truncated objects. Use an object store and client that commit whole objects (S3 PUT is atomic; avoid multipart flows that can leave partial keys on failure), and make sure flushes complete before the pod terminates:

ingester:
  wal:
    enabled: true
    flush_on_shutdown: true
  flush_op_timeout: 10m

Wipe the corrupt local index directory so the shipper re-downloads a clean copy from object storage. This is safe for the local cache because the authoritative index lives in the bucket:

kubectl exec -it loki-ingester-0 -- rm -rf /loki/tsdb-cache/*
kubectl rollout restart statefulset/loki-ingester

Verify schema_config store and object_store match the data on disk. If the data was written as boltdb-shipper and you now read it as TSDB (or vice versa), every read fails the magic-number check — align the schema period to how the data was actually written:

schema_config:
  configs:
    - from: 2023-06-01
      store: boltdb-shipper
      object_store: s3
      schema: v12
    - from: 2024-01-01
      store: tsdb
      object_store: s3
      schema: v13

What to watch out for

• invalid magic number 0 specifically signals an empty/truncated file — check object size first; a zero-byte key is the fastest confirmation.
• Deleting a corrupt object is only safe when replication_factor > 1 or you accept losing that chunk; without replicas, the data in that object is gone.
• Local boltdb/TSDB caches are disposable — wiping them forces a clean re-download — but never delete the authoritative index in the bucket.
• A schema/store mismatch produces this error across a whole time range, not just one object; if many keys fail together, suspect the schema, not corruption.
• Failing disks tend to corrupt more than one file over time — if local index corruption recurs, replace the volume rather than repeatedly wiping the cache.

Related

• Loki Error Guide: ‘failed to flush chunks’ — an interrupted flush is a common source of the truncated objects that fail the magic-number check.
• Loki Error Guide: ‘access denied’ to the object store — rule out a permissions failure before concluding an object is corrupt.
• Loki Error Guide: ‘context deadline exceeded’ — a timed-out read can look similar; use it to separate slow storage from corrupt data.