🚨 Loki Error Guide: 'no space left on device' on the WAL — Stop the Ingester Disk From Filling Up

Overview

Loki’s ingester writes incoming log entries to a write-ahead log on local disk before they are flushed to object storage. When that disk fills, the WAL write fails:

level=error msg="failed to write to WAL" err="write /loki/wal/00000123: no space left on device"

and the loki_ingester_wal_disk_full_failures_total counter starts incrementing. The WAL exists so unflushed in-memory chunks survive a restart — segments accumulate until their data is flushed to storage, at which point they are truncated and the space is reclaimed. A full WAL disk almost always means one of two things: the volume is too small for the ingest rate, or flushing has stalled so segments never get truncated and pile up. Left unaddressed, the ingester cannot durably record new logs and you risk data loss on the next restart. The fix is to relieve the disk and, crucially, to fix whatever is stopping segments from being truncated.

Symptoms

• Ingester logs repeat failed to write to WAL ... no space left on device.
• loki_ingester_wal_disk_full_failures_total climbs steadily.
• The WAL PVC shows near-100% usage; df on /loki/wal is full.
• New logs fail to ingest or are dropped, and the ingester may crash-loop on restart while replaying a large WAL.
• Frequently paired with failed to flush errors — the WAL grows because chunks are not reaching object storage.

Common Root Causes

• WAL PVC too small for the ingest volume — the disk was sized for a lower log rate and cannot hold segments long enough between flushes.
• Chunks are not flushing — storage errors (bad credentials, unreachable bucket, throttling) stall flushes, so WAL segments are never truncated and accumulate without bound.
• replay_memory_ceiling / checkpoint tuning — checkpoints run infrequently or the ceiling forces excessive on-disk retention, keeping more segments than necessary.
• Leftover old WAL segments — segments from a prior incident were never cleaned up and are consuming space alongside the live WAL.
• Undersized burst headroom — a traffic spike produced more WAL than the disk can hold before the flush cycle catches up.

How to diagnose

1. Confirm the disk is actually full and the failures counter is moving:

   kubectl logs -l app=loki,component=ingester --tail=200 | grep -i 'no space left'
   kubectl exec -it loki-ingester-0 -- df -h /loki/wal

2. Measure what the WAL directory is consuming and whether segments are being truncated or piling up:

   kubectl exec -it loki-ingester-0 -- du -sh /loki/wal
   kubectl exec -it loki-ingester-0 -- ls -la /loki/wal | head -20

3. Check whether flushes are failing — a stalled flush path is the most common reason the WAL cannot shrink:

   curl -s http://loki-ingester:3100/metrics \
     | grep -E 'loki_ingester_wal_disk_full_failures_total|failed|memory_chunks'
   kubectl logs -l app=loki,component=ingester --tail=300 | grep -i 'failed to flush'

4. Inspect the WAL configuration for dir, ceiling, and shutdown flush behaviour:

   curl -s http://loki:3100/config | grep -A8 'wal:'

5. Confirm the intended WAL settings match what the ingester is running:

   ingester:
     wal:
       enabled: true
       dir: /loki/wal
       replay_memory_ceiling: 4GB
       flush_on_shutdown: true

Fixes

Grow the WAL PVC so it comfortably holds segments across a full flush cycle plus burst headroom. Expand the volume (the StorageClass must allow expansion), then let the ingester resume:

# StatefulSet volumeClaimTemplate — increase the WAL request
volumeClaimTemplates:
  - metadata:
      name: wal
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 50Gi   # raised from 10Gi

Fix the underlying flush failures. If chunks are not reaching object storage, the WAL will refill no matter how large you make it — segments are only truncated once their data is flushed. Resolve the storage error first:

# Prove the ingester identity can write to the bucket, then watch memory drain
aws s3 cp /tmp/probe.txt s3://my-loki-bucket/probe.txt
curl -s localhost:3100/metrics | grep loki_ingester_memory_chunks

Tune ingester.wal so the directory and replay ceiling suit your ingest rate. A well-placed dir on a dedicated volume and a sane replay_memory_ceiling keep on-disk retention bounded:

ingester:
  wal:
    enabled: true
    dir: /loki/wal
    replay_memory_ceiling: 4GB
    flush_on_shutdown: true

Ensure checkpoints run so the WAL is periodically compacted and truncated rather than growing linearly. Verify the ingester is checkpointing and not erroring mid-checkpoint:

kubectl logs -l app=loki,component=ingester --tail=300 \
  | grep -iE 'checkpoint|truncat'

Monitor disk usage and alert before it fills. Page on WAL volume utilization and on loki_ingester_wal_disk_full_failures_total so you act on a filling disk rather than a full one:

# Prometheus alert: WAL disk pressure before it becomes fatal
- alert: LokiWALDiskFilling
  expr: increase(loki_ingester_wal_disk_full_failures_total[5m]) > 0
  for: 5m
  labels: { severity: critical }

What to watch out for

• Growing the PVC alone is a band-aid if flushes are failing — the enlarged disk simply fills more slowly; always check the flush path.
• Never delete the WAL directory to reclaim space while the ingester is serving; those segments hold unflushed logs and removing them loses data.
• A crash-loop on startup after the disk filled is often the WAL replay running out of space or memory — fix the disk and consider replay_memory_ceiling before restarting.
• loki_ingester_wal_disk_full_failures_total is the signal to watch; alert on it well before the volume hits 100%.
• Sizing the WAL for average ingest ignores bursts — leave headroom so a traffic spike does not fill the disk before the flush cycle catches up.

Related

• Loki Error Guide: ‘failed to flush chunks’ — stalled flushes are the top reason the WAL never truncates and the disk fills.
• Loki Error Guide: ‘entry out of order’ — another ingester write-path failure worth ruling out during a WAL incident.
• Loki Error Guide: ‘maximum active stream limit’ — high stream cardinality drives WAL and memory growth on the same ingesters.