🚨 Loki Error Guide: 'failed to flush chunks' — Fix Object Storage Writes and Ingester Backpressure

Overview

Loki ingesters buffer logs into chunks in memory and periodically flush them to object storage. When that flush fails, the ingester logs:

level=error msg="failed to flush" err="failed to flush chunks: store put chunk: <object-storage error>, num_chunks: 12"

with the underlying storage cause attached, such as:

failed to flush chunks: store put chunk: AccessDenied: Access Denied, status code: 403

Flushing is how Loki durably persists logs and frees ingester memory. If chunks cannot be written — bad credentials, an unreachable bucket, throttling, or a full disk on the WAL path — they stay pinned in memory, the flush queue backs up, and the ingester’s memory grows. Sustained flush failures risk OOM kills and data loss on restart. The message almost always wraps a specific object-storage or filesystem error that names the real problem.

Symptoms

• Ingester logs repeat failed to flush / failed to flush chunks with a storage error attached.
• Ingester memory (loki_ingester_memory_chunks) climbs steadily and pods approach their limits or OOM.
• Recent logs are queryable (still in memory) but missing after an ingester restart.
• loki_ingester_chunk_age_seconds and the flush queue length grow.
• The wrapped error names the cause: 403 AccessDenied, NoSuchBucket, connection refused, no space left on device, or SlowDown.

Common Root Causes

• Bad or expired storage credentials — IAM role/keys lacking s3:PutObject, producing 403 AccessDenied.
• Wrong bucket or endpoint — NoSuchBucket, DNS failure, or a misconfigured S3-compatible endpoint.
• Network/connectivity loss to object storage — connection refused, timeouts, TLS errors.
• Storage throttling — S3 SlowDown / 503 under high write rate.
• Full or failing WAL/filesystem on the ingester — no space left on device.
• Flush queue too small for the chunk production rate, so flushes fall behind.

Diagnostic Workflow

Read the wrapped error first — it names the cause. Then confirm ingester health from metrics:

journalctl -u loki --since '15 min ago' | grep -i 'failed to flush'
curl -s http://loki-ingester:3100/metrics \
  | grep -E 'loki_ingester_memory_chunks|loki_ingester_chunk_age_seconds|loki_ingester_.*flush'

Verify the storage config and credentials block:

storage_config:
  aws:
    s3: s3://us-east-1/my-loki-bucket
    endpoint: s3.us-east-1.amazonaws.com
    s3forcepathstyle: false
common:
  storage:
    s3:
      bucket_names: my-loki-bucket
      region: us-east-1
      access_key_id: ${AWS_ACCESS_KEY_ID}
      secret_access_key: ${AWS_SECRET_ACCESS_KEY}

Test bucket access with the same identity the ingester uses:

# Prove PutObject works for the ingester's credentials/role
aws s3 cp /tmp/probe.txt s3://my-loki-bucket/probe.txt
aws s3 ls s3://my-loki-bucket/

If storage is healthy, check the flush queue and chunk lifecycle settings for backpressure:

ingester:
  flush_op_timeout: 10m
  chunk_idle_period: 30m
  max_chunk_age: 2h
  chunk_target_size: 1572864
  concurrent_flushes: 32     # raise if the flush queue backs up

Example Root Cause Analysis

After migrating Loki to a new EKS cluster, ingesters logged failed to flush chunks: store put chunk: AccessDenied: 403 every flush cycle. Queries of recent logs worked, so the write path from clients was fine — but nothing was reaching S3. Memory chunks climbed until pods neared their limit.

An aws s3 cp using the ingester’s IRSA role reproduced the 403: the new cluster’s service-account IAM role had s3:ListBucket but was missing s3:PutObject on the bucket. Adding s3:PutObject/s3:GetObject/s3:DeleteObject to the role policy fixed flushes immediately; memory chunks drained as the backlog wrote out. As a safety net they raised concurrent_flushes to handle the catch-up burst and confirmed the WAL was enabled so any future flush outage would replay on restart instead of losing data. The root cause was purely IAM — the wrapped 403 AccessDenied had pointed straight at it.

Prevention Best Practices

• Grant ingesters the full object-storage permission set (PutObject, GetObject, DeleteObject, ListBucket) and verify with a real write test.
• Use IAM roles / workload identity over static keys so credentials do not silently expire.
• Enable the WAL so unflushed chunks replay after a restart instead of being lost.
• Monitor loki_ingester_memory_chunks and flush failure rate; alert before memory pressure becomes an OOM.
• Size concurrent_flushes and flush_op_timeout for peak chunk production and storage latency.
• Handle storage throttling (SlowDown) with retries and, if persistent, request higher bucket throughput.

Quick Command Reference

# The wrapped cause and ingester memory pressure
journalctl -u loki | grep 'failed to flush'
curl -s localhost:3100/metrics | grep loki_ingester_memory_chunks

# Prove the ingester's identity can write to the bucket
aws s3 cp /tmp/probe.txt s3://my-loki-bucket/probe.txt

# Confirm storage and flush settings
curl -s http://loki:3100/config | grep -A4 -E 'storage|concurrent_flushes'

ingester:
  concurrent_flushes: 32
  flush_op_timeout: 10m
  wal:
    enabled: true

Conclusion

failed to flush chunks means Loki cannot persist buffered logs to object storage, and the wrapped error — 403 AccessDenied, NoSuchBucket, connection refused, no space left on device — names the real cause. Start there: fix storage credentials, bucket, and connectivity, then relieve backpressure by tuning concurrent_flushes and chunk-age settings. Keep the WAL enabled and alert on ingester memory so a flush outage means a replayable delay, not lost logs or an OOM-killed ingester.