🚨 Loki Error Guide: 'rpc error: code = Unavailable desc = transport is closing' — Stabilize gRPC Between Components

Overview

Loki components talk to each other over gRPC. When one side tears down a connection in the middle of a call, the caller logs:

rpc error: code = Unavailable desc = transport is closing

The Unavailable code means the RPC never completed because the underlying HTTP/2 transport went away — a distributor calling an ingester, a querier calling an ingester, or the query frontend calling a querier suddenly lost its connection mid-request. This is almost always a connection lifecycle problem, not a Loki logic bug: the target pod restarted, a keepalive probe killed an idle connection, a node blipped, or a load balancer reset a long-lived HTTP/2 stream. The fix is to find what closed the transport and stop it from happening under normal load.

Symptoms

• Callers (distributor, querier, frontend) log transport is closing in bursts, then recover.
• Errors line up with ingester or querier pod restarts, OOMKills, or rescheduling events.
• Long-idle connections fail on first reuse after a quiet period.
• A gRPC load balancer or service mesh resets connections at a fixed interval.
• Writes or queries fail intermittently but succeed on retry, pointing at transient connection loss.

Common Root Causes

• Target pod restarted, OOMKilled, or rescheduled — the ingester or querier on the other end went away mid-call, closing the transport.
• gRPC keepalive killed an idle connection — mismatched keepalive_time/keepalive_timeout between client and server closed a connection the caller still expected to use.
• Node or network blip — a brief partition or node pressure event severed the HTTP/2 connection.
• An L4 load balancer resetting long-lived HTTP/2 — routing gRPC through a connection-terminating LB instead of direct pod-to-pod causes periodic resets.
• Server enforcing a minimum keepalive interval — the ingester’s gRPC server rejects clients that ping too aggressively and closes the connection with a GOAWAY.

How to diagnose

1. Check the target for restarts and OOMKills — the most common cause is the callee simply going away:

   kubectl get pods -l app=loki,component=ingester
   kubectl describe pod loki-ingester-1 | grep -A6 'Last State\|OOMKilled\|Reason'

2. Correlate error bursts with pod events — line up the log timestamps with restart/reschedule events:

   kubectl get events --sort-by=.lastTimestamp | grep -i 'loki\|OOM\|Killing'

3. Confirm gRPC reachability to the target — verify the port is open when the pod is up:

   kubectl exec -it deploy/loki-querier -- nc -vz loki-ingester-1.loki-headless 9095

4. Inspect the current gRPC keepalive config — mismatched client/server keepalive is a frequent culprit:

   ingester_client:
     grpc_client_config:
       keepalive_time: 30s
       keepalive_timeout: 10s

5. Check whether gRPC traffic crosses an L4 load balancer — confirm components resolve each other via a headless service, not a connection-terminating LB:

   kubectl get svc loki-headless -o yaml | grep -i 'clusterIP\|None'

Fixes

Stop the target pods from dying by fixing the underlying restart or OOMKill. Raise memory limits so ingesters and queriers are not OOMKilled mid-call:

resources:
  requests:
    memory: 4Gi
  limits:
    memory: 6Gi   # headroom so the ingester is not OOMKilled under load

Tune gRPC client keepalive so idle connections are probed and kept alive consistently on both ends, and clients do not ping more often than the server allows:

ingester_client:
  grpc_client_config:
    keepalive_time: 30s
    keepalive_timeout: 10s
frontend_worker:
  grpc_client_config:
    keepalive_time: 30s
    keepalive_timeout: 10s

Route gRPC pod-to-pod via a headless service so connections are not reset by an L4 load balancer terminating HTTP/2. Point components at a ClusterIP: None service:

apiVersion: v1
kind: Service
metadata:
  name: loki-headless
spec:
  clusterIP: None            # headless: direct pod-to-pod gRPC, no LB reset
  selector: { app: loki }
  ports:
    - name: grpc
      port: 9095

Add client-side retries so a transient transport is closing is retried transparently instead of surfacing as a failed request:

querier:
  ingester_client:
    grpc_client_config:
      max_recv_msg_size: 104857600
  # queriers retry ingesters on transient Unavailable; keep RF>=2 so a
  # single ingester restart does not lose the write

Relax the server keepalive-enforcement policy if the server is sending GOAWAY because clients ping too aggressively — align min_time with the client keepalive_time:

server:
  grpc_server_min_time_between_pings: 10s
  grpc_server_ping_without_stream_allowed: true

What to watch out for

• Unavailable / transport is closing is a connection-lifecycle signal, not a Loki bug — chase what closed the transport (restart, OOM, keepalive, LB) rather than tweaking query logic.
• Client and server keepalive settings must be compatible: if the client pings faster than the server’s minimum, the server closes the connection and you get more of exactly this error.
• Routing gRPC through an L4 load balancer that terminates HTTP/2 causes periodic, hard-to-explain resets; direct pod-to-pod via a headless service avoids it.
• With replication_factor of 1, a single ingester restart turns this transient error into real data loss on the affected pushes; keep RF at 2 or more.
• Occasional bursts during a controlled rollout are expected; sustained bursts with no matching pod events point at keepalive or load-balancer settings.

Related

• Loki Error Guide: ‘empty ring’ — when repeated component loss escalates into ring-membership failure.
• Loki Error Guide: ‘context canceled’ — the cancellation counterpart when a call is aborted rather than the transport dropped.
• Loki Error Guide: ‘grpc: received message larger than max’ — a different gRPC failure driven by message size rather than connection loss.