🚨 Loki Error Guide: 'InternalError: We encountered an internal error status code: 500' — Retry, Then Alert on Sustained Failures

Overview

Loki logs this error when the object store hits a fault on its own side and returns a 500, asking the client to try again:

InternalError: We encountered an internal error, please try again.
	status code: 500, request id=..., host id=...

Ingesters see it on flush, the compactor on compaction, and queriers on reads. Unlike a 403 or 404, a 500 is provider-side: the request was valid and authenticated, but the store failed transiently. These are almost always fixed by a retry, and the store explicitly says so. The engineering task is not to eliminate them — you cannot — but to ensure Loki retries with backoff and to alert only when the failures are sustained rather than momentary.

Symptoms

• Ingesters log occasional InternalError ... status code: 500 that succeed on the next attempt, with no user-visible impact.
• Config, credentials, bucket, and region are all correct and unchanged, ruling out 403/404 causes.
• Brief bursts of 500s correlate with a provider status-page incident.
• loki_ingester_memory_chunks stays flat because retries clear the backlog quickly.
• Sustained 500s (not brief bursts) coincide with rising flush latency and a growing chunk backlog.

Common Root Causes

• Transient provider-side fault — the store had a momentary internal error unrelated to Loki’s request.
• Retries disabled or too shallow — with max_retries low or zero, a single 500 fails the operation instead of succeeding on the next try.
• Provider incident — sustained 500s indicate an ongoing outage or degradation on the storage backend.
• Overloaded self-hosted store — on MinIO/Ceph, an under-provisioned gateway or backend can emit 500s under load.
• Backoff not applied — retrying instantly without backoff can amplify load during a wobble and prolong the 500s.

How to diagnose

1. Confirm the status and whether it is transient by grepping the logs and checking that retries succeed:

   kubectl logs -l app=loki,component=ingester --since=15m \
     | grep -iE 'InternalError|status code: 500|internal error'

2. Distinguish a burst from sustained failure by looking at S3 5xx error metrics over time:

   aws cloudwatch get-metric-statistics \
     --namespace AWS/S3 --metric-name 5xxErrors \
     --dimensions Name=BucketName,Value=loki-chunks-prod \
     --start-time 2026-07-12T00:00:00Z --end-time 2026-07-12T01:00:00Z \
     --period 60 --statistics Sum

3. Check that retries are configured in the running Loki config:

   kubectl exec -it deploy/loki-ingester -- \
     wget -qO- http://localhost:3100/config | grep -A6 'aws:'

4. Watch flush health to see whether the 500s are actually backing up ingestion:

   kubectl exec -it deploy/loki-ingester -- \
     wget -qO- http://localhost:3100/metrics | grep -i 'loki_ingester_memory_chunks'

5. Rule out a provider incident by checking the backend’s status and, for self-hosted stores, the gateway health:

   # example alert to separate a burst from a sustained provider fault
   - alert: LokiObjectStore5xxSustained
     expr: increase(loki_ingester_chunk_age_seconds_sum[5m]) > 0
       and rate(cortex_dynamo_failures_total[5m]) > 0
     for: 10m

Fixes

Ensure retries with backoff are enabled so a transient 500 is absorbed rather than failing the flush. Set a sensible max_retries and reasonable HTTP timeouts:

storage_config:
  aws:
    s3: s3://us-east-1/loki-chunks-prod
    bucketnames: loki-chunks-prod
    region: us-east-1
    s3forcepathstyle: false
    max_retries: 5
    http_config:
      idle_conn_timeout: 90s
      response_header_timeout: 30s

Treat brief bursts as expected and do not page on a single 500. Scope alerts so momentary provider wobbles that retries clear never wake anyone:

- alert: LokiObjectStoreInternalErrorBurst
  expr: rate(loki_request_duration_seconds_count{status_code="500"}[5m]) > 0
  for: 15m           # only fire when it persists, not on a blip
  labels: { severity: warning }

Alert on sustained failures via flush metrics so a real provider outage — where retries stop clearing — is caught. Watch chunk age and flush failures rather than raw 500 counts:

- alert: LokiFlushBackingUp
  expr: loki_ingester_memory_chunks > 500000
    and increase(loki_ingester_chunk_age_seconds_sum[10m]) > 0
  for: 10m
  labels: { severity: critical }
  annotations:
    summary: "Loki flushes backing up; object store may be returning sustained 500s"

Check the provider status page (or gateway) when 500s persist — if retries no longer clear the backlog, the fault is on the store, not Loki. For self-hosted MinIO/Ceph, scale or repair the gateway; for a managed provider, confirm an incident and let backoff ride it out:

kubectl exec -it deploy/loki-ingester -- sh -c \
  'for i in 1 2 3; do aws s3 ls s3://loki-chunks-prod/ && break || sleep 2; done'

What to watch out for

• A 500 is provider-side and valid to retry; do not chase IAM, bucket, or region fixes for InternalError the way you would a 403 or 404.
• Retrying without backoff can amplify load during a store wobble; make sure max_retries is paired with exponential backoff, not instant retries.
• Alerting on individual 500s causes fatigue — they happen at scale routinely. Alert on sustained failure signals (flush backlog, chunk age), not raw counts.
• A brief burst that clears is expected; a sustained rise that does not clear is a provider incident and needs escalation, not more tuning.
• On self-hosted stores, persistent 500s usually mean the gateway or backend is overloaded; scale it rather than only tuning Loki’s retries.

Related

• Loki Error Guide: ‘failed to flush chunks’ — the flush-path symptom that sustained 500s produce.
• Loki Error Guide: ‘context deadline exceeded’ — object-store timeouts that overlap with provider-side faults.
• Loki Error Guide: ‘AccessDenied: Access Denied status code: 403’ — the permissions 403 to rule out so you do not misread a transient 500.