🚨 Loki Error Guide: 'failed to compact files' — Get the Compactor Running and Retention Working

Overview

The compactor merges per-ingester index files into a single compact index per period and applies retention by deleting expired data. When a compaction cycle cannot complete, it logs:

level=error msg="failed to compact files" err="..."

sometimes surfaced as the broader:

level=error msg="failed to run compaction" err="..."

The wrapped error names the cause, and it is usually one of a short list: the compactor lacks object-store permissions (retention needs s3:DeleteObject), more than one compactor is running, its working_directory is out of disk, or the schema does not match the index format it is trying to compact. When compaction stalls, the index stops shrinking and retention stops deleting expired data — so storage grows and queries slow as they scan un-compacted index files. The compactor is a singleton by design; most incidents come from violating that or from missing delete permissions.

Symptoms

• Compactor logs repeat failed to compact files / failed to run compaction with a wrapped cause.
• Retention is not enforced — data past the retention period is still present and queryable, and storage keeps growing.
• The index store accumulates many small per-ingester index files that are never merged.
• Queries slow down over time as they scan an un-compacted, fragmented index.
• The wrapped error names it: AccessDenied on delete, no space left on device on the working directory, or a schema/format complaint.

Common Root Causes

• Object store missing s3:DeleteObject — retention deletes expired chunks and index files; without delete permission, retention-driven compaction fails.
• More than one compactor running — the compactor must be exactly one instance; two of them race on the same index and corrupt or fail each other’s work.
• compactor.working_directory out of disk — compaction downloads and rewrites index files locally; a full working directory aborts the cycle.
• Index format / schema mismatch — the schema_config does not match the on-disk index (boltdb-shipper vs TSDB), so the compactor cannot parse the files.
• Retention misconfigured — retention_enabled set without a valid shared_store, delete-request store, or period, so the retention step errors.

How to diagnose

1. Read the wrapped error — it points straight at permissions, disk, or schema:

   kubectl logs -l app=loki,component=compactor --tail=200 \
     | grep -iE 'failed to compact|failed to run compaction'

2. Confirm exactly one compactor is running — this is the single most common misconfiguration:

   kubectl get pods -l app=loki,component=compactor
   # There must be exactly ONE. Scale a Deployment to replicas: 1.

3. Check the working directory’s disk on the compactor pod:

   kubectl exec -it loki-compactor-0 -- df -h /loki/compactor
   kubectl exec -it loki-compactor-0 -- du -sh /loki/compactor

4. Test object-store permissions, including delete, with the compactor’s identity:

   aws s3 cp /tmp/probe.txt s3://my-loki-bucket/probe.txt   # Put
   aws s3 rm s3://my-loki-bucket/probe.txt                   # Delete (retention needs this)

5. Verify the compactor and schema configuration line up with the on-disk index format:

   compactor:
     working_directory: /loki/compactor
     shared_store: s3
     retention_enabled: true
     compaction_interval: 10m

Fixes

Run exactly one compactor instance. Two compactors racing on the same shared index is a guaranteed failure. Pin the Deployment to a single replica (or run the single-binary compactor target once):

# Compactor Deployment — one and only one replica
apiVersion: apps/v1
kind: Deployment
metadata:
  name: loki-compactor
spec:
  replicas: 1

Grant the compactor full object-store permissions including delete. Retention deletes expired objects, so s3:DeleteObject is mandatory alongside get/put/list:

{
  "Effect": "Allow",
  "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket"],
  "Resource": [
    "arn:aws:s3:::my-loki-bucket",
    "arn:aws:s3:::my-loki-bucket/*"
  ]
}

Give working_directory enough disk. Compaction stages index files locally; if that volume is full the cycle aborts. Point it at a sized volume and confirm headroom:

compactor:
  working_directory: /loki/compactor   # on a PVC with room to stage the index
  shared_store: s3

Set retention correctly — enable it with a valid interval and delay so the retention step does not error. Keep retention configured in the compactor only, never in multiple components:

compactor:
  working_directory: /loki/compactor
  shared_store: s3
  retention_enabled: true
  retention_delete_delay: 2h
  compaction_interval: 10m
limits_config:
  retention_period: 744h

Match schema_config to the on-disk index format. If the data was written with boltdb-shipper and the compactor expects TSDB (or the reverse), it cannot parse the files — align the schema periods to how the index was actually written:

schema_config:
  configs:
    - from: 2023-06-01
      store: boltdb-shipper
      object_store: s3
      schema: v12
      index:
        prefix: index_
        period: 24h
    - from: 2024-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

What to watch out for

• The compactor is a strict singleton — two instances do not share load, they corrupt each other’s work; enforce replicas: 1.
• Retention silently does nothing if delete permission is missing, so storage grows unnoticed until you notice the bill; test s3 rm with the compactor’s identity.
• Configure retention in the compactor only; enabling it in multiple places causes inconsistent deletes and the dangling-index-reference problems seen elsewhere.
• A schema mismatch fails every compaction cycle, not just one — if compaction never succeeds after a store migration, check schema_config first.
• The working_directory needs real disk headroom; a compactor that worked at low volume can start failing as the index grows and staging needs more space.

Related

• Loki Error Guide: ‘access denied’ to the object store — the missing s3:DeleteObject that breaks retention-driven compaction.
• Loki Error Guide: ‘failed to flush chunks’ — the same object-store permission and connectivity issues that stall the write path also stall compaction.
• Loki Error Guide: ‘too many chunks to fetch’ — a fragmented, un-compacted index makes queries fetch more, so fixing compaction relieves it.