🚨 Loki Error Guide: 'failed to replay WAL' — Recover an Ingester From a Corrupt Write-Ahead Log

Overview

When an ingester restarts, it replays its write-ahead log to rebuild the in-memory chunks that had not yet been flushed. If the WAL is damaged, the replay logs:

level=error msg="failed to replay WAL" err="..."

often followed by a recovery notice:

level=warn msg="encountered corruption while replaying WAL, recovering what we can"

Corruption almost always sits in the last segment, because that is the one being written when the process died. An unclean shutdown, a SIGKILL, an OOM, or a node crash can leave that final segment half-written. Loki is built for this: it truncates the corrupt tail, recovers every intact segment before it, and continues — accepting the loss of whatever in-memory data had not been checkpointed or flushed. Your job is usually to let that recovery happen, then remove the conditions that cause unclean shutdowns so it does not repeat. Only a persistent replay loop calls for manual intervention.

Symptoms

• On startup the ingester logs failed to replay WAL and/or encountered corruption while replaying WAL, recovering what we can.
• The ingester eventually comes up, but a small window of the most recent unflushed logs is missing.
• The corruption always maps to the final WAL segment, not the middle of the log.
• The incident follows a node crash, OOM kill, SIGKILL, or a pod terminated before it finished shutting down.
• In the worst case the ingester crash-loops, unable to get past replay at all.

Common Root Causes

• Unclean shutdown / SIGKILL before checkpoint — the pod was force-killed (short termination grace period, node drain, kill -9) before it flushed or checkpointed, leaving a partial segment.
• Disk fault — a failing or full volume corrupted a WAL segment mid-write.
• Partially-written last segment after a node crash or OOM — the kernel or scheduler killed the process while it was appending, truncating the active segment.
• No graceful termination window — terminationGracePeriodSeconds too short for the ingester to flush on shutdown, so it is killed mid-write every restart.
• Repeated OOM kills — an under-resourced ingester is killed again and again, each time leaving a corrupt tail.

How to diagnose

1. Read the replay logs to see whether recovery succeeded or the ingester is stuck looping:

   kubectl logs loki-ingester-0 --tail=200 \
     | grep -iE 'replay WAL|corruption|recovering what we can'

2. Check whether the pod is recovering or crash-looping — a healthy recovery reaches Ready; a loop restarts repeatedly:

   kubectl get pod loki-ingester-0 -o wide
   kubectl describe pod loki-ingester-0 | grep -iE 'OOMKilled|Last State|Reason|Restart Count'

3. Inspect the WAL directory and its disk to spot a truncated final segment or a full/failing volume:

   kubectl exec -it loki-ingester-0 -- ls -la /loki/wal
   kubectl exec -it loki-ingester-0 -- df -h /loki/wal

4. Confirm the shutdown path — grace period and flush_on_shutdown — that determines whether restarts are clean:

   kubectl get statefulset loki-ingester -o yaml \
     | grep -iE 'terminationGracePeriodSeconds'

5. Verify the WAL and replication settings against what you intend to run:

   ingester:
     wal:
       enabled: true
       dir: /loki/wal
       flush_on_shutdown: true
   common:
     replication_factor: 3

Fixes

Let Loki recover automatically. The recovering what we can message means it is doing the right thing — truncating the corrupt tail and restoring earlier segments. Expect to lose only the unflushed, un-checkpointed tail. Do not delete anything; let the ingester finish and reach Ready:

kubectl logs -f loki-ingester-0 | grep -iE 'recovering|WAL replay|is now'

Enable flush_on_shutdown and a proper termination grace period so restarts flush in-memory chunks cleanly instead of leaving a partial segment. Give the ingester enough time to drain:

ingester:
  wal:
    enabled: true
    flush_on_shutdown: true
# StatefulSet: allow time to flush before SIGKILL
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 300

Break a replay loop by removing the corrupt WAL directory — as a last resort. If the ingester cannot get past replay after repeated restarts, wiping the WAL lets it start clean, at the cost of the unflushed in-memory data it held:

# LAST RESORT: loses unflushed data on this ingester only
kubectl exec -it loki-ingester-0 -- rm -rf /loki/wal/*
kubectl delete pod loki-ingester-0   # restarts clean

Rely on replication_factor so a peer still has the data. With replication of 3, the entries the crashed ingester lost were also written to other ingesters, so recovery does not mean losing those logs from the cluster:

common:
  replication_factor: 3
ingester:
  lifecycler:
    ring:
      replication_factor: 3

Address the root shutdown cause — right-size memory to stop OOM kills, and avoid force-killing pods. If the WAL keeps getting corrupted, the process keeps dying mid-write; fixing that stops the recurrence:

kubectl describe pod loki-ingester-0 | grep -i OOMKilled
kubectl set resources statefulset/loki-ingester -c ingester --limits=memory=8Gi

What to watch out for

• Corruption in the last segment is expected after a crash and is safely recoverable; corruption spread across many segments points at a failing disk, not a normal unclean shutdown.
• Deleting the WAL directory always loses that ingester’s unflushed in-memory data — only do it to break a genuine replay loop, and only when replication covers the data elsewhere.
• A short terminationGracePeriodSeconds guarantees this error on every rollout — give the ingester time to flush on shutdown.
• Repeated OOM kills produce repeated WAL corruption; treat the OOM as the real bug, not the replay message.
• Without replication_factor > 1, recovering from a corrupt WAL means genuinely losing those logs — replication is what makes recovery non-destructive at the cluster level.

Related

• Loki Error Guide: ‘failed to flush chunks’ — flush health determines how much unflushed data a WAL replay can lose.
• Loki Error Guide: ‘empty ring’ — ingester ring problems often accompany crash-looping ingesters during recovery.
• Loki Error Guide: ‘entry out of order’ — another ingester write-path error to distinguish from WAL replay failures.