🚨 Loki Error Guide: '502 Bad Gateway' — Recover the Gateway-to-Loki Upstream

Overview

A reverse proxy or load balancer sits in front of Loki (commonly the loki-gateway nginx). When that proxy cannot get a valid response from its Loki upstream, it returns:

502 Bad Gateway

In Grafana the same failure surfaces as:

Loki: Bad Gateway. 502

A 502 is emitted by the gateway, not by Loki — it means the proxy forwarded the request but the upstream Loki pod refused the connection, closed it, or returned something malformed. Either the upstream is down or not ready, the gateway is pointed at the wrong service or port, or the response exceeded the gateway’s buffers/timeouts. Because the error originates at the proxy, you diagnose it from both sides: confirm the Loki pod is healthy, then confirm the gateway is forwarding to it correctly.

Symptoms

• Grafana shows Loki: Bad Gateway. 502 on queries or the datasource test.
• The gateway (nginx) access log records 502 with a Loki upstream address.
• 502s appear right after a Loki pod crash, OOMKill, or restart.
• The gateway is up and reachable, but every request to Loki fails.
• 502s clear on their own once the upstream finishes WAL replay and becomes ready.

Common Root Causes

• Upstream Loki pod crashed, OOMKilled, or not ready — the gateway has nothing healthy to forward to, so it returns 502.
• Gateway upstream points at the wrong port or service — proxy_pass targets a name/port that is not the live Loki HTTP endpoint on 3100.
• Response larger than the gateway buffer — a big query response overflows nginx proxy buffers and the connection is dropped as a 502.
• Upstream still starting — Loki is replaying its WAL and not yet serving on 3100, so the gateway cannot connect.
• Readiness not gated — the gateway forwards to a pod that has not passed its readiness probe, hitting a port that is not accepting requests yet.

How to diagnose

1. Check the upstream Loki pod health and readiness — a crashed or not-ready pod is the most common cause:

   kubectl get pods -l app=loki
   kubectl describe pod loki-0 | grep -A6 'Last State\|OOMKilled\|Readiness'

2. Read the gateway access/error log — confirm the 502 and see which upstream it tried:

   kubectl logs -l app=loki,component=gateway --tail=100 | grep 502

3. Curl Loki directly, bypassing the gateway — prove whether Loki itself is serving on 3100:

   kubectl exec -it deploy/loki-gateway -- \
     curl -s -o /dev/null -w '%{http_code}\n' http://loki:3100/ready

4. Verify the gateway upstream config — confirm proxy_pass targets the right service and port:

   # loki-gateway nginx.conf (upstream block)
   upstream loki {
     server loki:3100;   # must be the Loki HTTP port
   }
   location / {
     proxy_pass http://loki;
   }

5. Check for WAL replay in progress — a starting ingester is not ready until replay finishes:

   kubectl logs loki-0 | grep -i 'replay\|WAL\|starting'

Fixes

Restore upstream Loki health and resources so the gateway has a ready pod to forward to. Fix the crash cause and give the pod enough memory to avoid OOMKills:

resources:
  requests:
    memory: 4Gi
  limits:
    memory: 6Gi   # avoid OOMKills that leave the gateway with no upstream

Correct the gateway upstream so proxy_pass points at the live Loki service on port 3100:

# loki-gateway nginx.conf
upstream loki {
  server loki:3100;
}
server {
  listen 8080;
  location / {
    proxy_pass         http://loki;
    proxy_http_version 1.1;
    proxy_set_header   Connection "";
  }
}

Raise proxy buffers and timeouts so large query responses and long queries do not overflow or time out into a 502:

proxy_read_timeout    310s;
proxy_send_timeout    310s;
proxy_buffer_size     16k;
proxy_buffers         8 32k;
proxy_busy_buffers_size 64k;

Wait out WAL replay and gate on readiness so the gateway does not forward to a pod that is still starting. Point the readiness probe at /ready:

readinessProbe:
  httpGet:
    path: /ready
    port: 3100
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 12   # allow time for WAL replay before ready

Add readiness gating at the service level so only ready Loki pods receive traffic — set publishNotReadyAddresses: false so not-ready pods stay out of the gateway’s upstream endpoints:

apiVersion: v1
kind: Service
metadata:
  name: loki
spec:
  publishNotReadyAddresses: false   # keep not-ready pods out of rotation
  selector: { app: loki }
  ports:
    - { name: http, port: 3100 }

What to watch out for

• A 502 is emitted by the gateway, not Loki — always confirm from both sides: is the upstream healthy, and is the gateway pointed at it correctly.
• Bypass the gateway with a direct curl to :3100/ready early; it instantly tells you whether the fault is Loki or the proxy layer.
• WAL replay can take minutes on a large ingester; 502s during startup are expected and clear once the pod is ready, so gate traffic on /ready rather than restarting.
• Undersized proxy buffers turn large but valid query responses into 502s; raise buffers before assuming the upstream is broken.
• A gateway upstream pointing at the wrong port or a not-ready pod produces a steady 502 even though Loki itself is perfectly healthy.

Related

• Loki Error Guide: ‘context deadline exceeded’ — the timeout that turns into a 502 when the gateway gives up on a slow upstream.
• Loki Error Guide: ‘too many outstanding requests’ — read-path overload behind the gateway that can present as gateway errors.
• Loki Error Guide: ‘empty ring’ — an upstream ring failure that leaves the gateway with nothing healthy to serve.