🚨 Loki Error Guide: 'SlowDown: Please reduce your request rate status code: 503' — Throttle and Cache the S3 Path

Overview

Loki logs this error when the object store throttles it for issuing requests faster than the store (or a hot prefix) will accept, returning a 503:

SlowDown: Please reduce your request rate.
	status code: 503, request id=..., host id=...

Ingesters see it on high-volume flushes, and queriers see it when heavy reads fan out to many objects. S3 scales per-prefix request rate, so a burst of PUTs to one hot prefix, a flood of GETs from uncached queries, or a storm of tiny chunks will trip throttling. The store expects the client to back off and retry; if Loki’s retries and caches are not tuned, the 503s cascade into failed flushes and slow queries.

Symptoms

• Ingesters log SlowDown ... status code: 503 in bursts during high ingest, and some flushes fail before succeeding on retry.
• Query latency spikes when panels fan out to many small chunks, each a separate uncached GET.
• loki_ingester_memory_chunks briefly climbs during throttling as flushes back up.
• The rate of S3 requests is high relative to data volume, indicating many small objects rather than few large ones.
• Disabling caches (or a cache outage) coincides with the onset of the 503s.

Common Root Causes

• High PUT/GET rate against a hot prefix — chunk and index keys concentrate on one prefix, exceeding its per-prefix request budget.
• Too many small chunks — low chunk_target_size/short chunk_idle_period produce a storm of tiny objects and one PUT each.
• Caching disabled or missing — no chunk or index cache means every query re-reads S3, hammering GET.
• Aggressive flush cadence — frequent flushes multiply PUT volume beyond what the prefix tolerates.
• No or shallow retry/backoff — without retries the first 503 fails the operation instead of succeeding on a short backoff.

How to diagnose

1. Confirm throttling and its cadence from the logs:

   kubectl logs -l app=loki,component=ingester --since=15m \
     | grep -iE 'SlowDown|status code: 503|reduce your request rate'

2. Check S3 request metrics to see whether request rate is high relative to bytes moved (many small objects):

   aws cloudwatch get-metric-statistics \
     --namespace AWS/S3 --metric-name AllRequests \
     --dimensions Name=BucketName,Value=loki-chunks-prod \
     --start-time 2026-07-12T00:00:00Z --end-time 2026-07-12T01:00:00Z \
     --period 60 --statistics Sum

3. Confirm whether caches are wired up by querying the running config:

   kubectl exec -it deploy/loki-querier -- \
     wget -qO- http://localhost:3100/config | grep -A5 chunk_store_config

4. Measure chunk size to detect a small-chunk storm — inspect the current chunk settings:

   ingester:
     chunk_target_size: 1572864   # ~1.5MB target; too-low values make many small chunks
     chunk_idle_period: 30m

5. Check cache hit ratio so you know reads are actually being served from cache rather than S3:

   kubectl exec -it deploy/loki-querier -- \
     wget -qO- http://localhost:3100/metrics | grep -i cache_hits

Fixes

Raise S3 retries and backoff so transient throttling is absorbed instead of failing the operation. Increase max_retries and let the SDK back off before giving up:

storage_config:
  aws:
    s3: s3://us-east-1/loki-chunks-prod
    bucketnames: loki-chunks-prod
    region: us-east-1
    s3forcepathstyle: false
    max_retries: 5
    http_config:
      idle_conn_timeout: 90s
      response_header_timeout: 30s

Add chunk and index caches so repeat reads never touch S3, cutting GET volume dramatically. Point chunk_store_config at a memcached tier:

chunk_store_config:
  chunk_cache_config:
    memcached:
      batch_size: 256
      parallelism: 10
    memcached_client:
      host: memcached-chunks.logging.svc
      service: memcached
      consistent_hash: true

Write fewer, larger chunks so each flush produces bigger objects and far fewer PUTs. Raise chunk_target_size and lengthen chunk_idle_period so chunks fill before they cut:

ingester:
  chunk_target_size: 2097152   # ~2MB, fewer/larger objects
  chunk_idle_period: 1h
  max_chunk_age: 2h

Spread key prefixes so requests are not all funnelled through one hot prefix. On providers/versions that support it, avoid a single shared prefix and let index/chunk keys distribute; on MinIO/Ceph, scale the gateway and confirm path-style is set:

storage_config:
  aws:
    s3: s3://us-east-1/loki-chunks-prod
    bucketnames: loki-chunks-prod
    region: us-east-1
    s3forcepathstyle: false
    max_retries: 5

What to watch out for

• Retries alone only mask throttling; if request rate stays high you will keep hitting 503s. Pair retries with caching and larger chunks to cut the underlying rate.
• A too-low chunk_target_size is a silent multiplier: it creates many tiny objects and one PUT each, inflating request rate far beyond your data volume.
• A cache outage can suddenly reintroduce SlowDown on the read path; alert on cache availability and hit ratio, not just on the 503s.
• Lengthening chunk_idle_period trades a little query freshness for far fewer objects; keep it within your acceptable ingest-to-query latency.
• On S3-compatible stores (MinIO/Ceph) the throttle usually reflects gateway capacity, not a per-prefix limit; scale the gateway rather than only tuning Loki.

Related

• Loki Error Guide: ‘failed to flush chunks’ — the flush-path symptom that sustained throttling produces.
• Loki Error Guide: ‘too many chunks to fetch’ — the read-side cost of many small chunks that also drives GET pressure.
• Loki Error Guide: ‘context deadline exceeded’ — object-store timeouts that throttling and retries can push you into.