🚨 Loki Error Guide: 'ResourceExhausted: grpc: received message larger than max' — Raise gRPC Message Size Limits

Overview

Loki components communicate over gRPC, and gRPC caps the size of a single message. When a frame exceeds that cap, the receiver aborts with a ResourceExhausted status:

rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5242880 vs. 4194304)

The sending side shows the mirror-image form:

rpc error: code = ResourceExhausted desc = trying to send message larger than max (6291456 vs. 4194304)

Every gRPC endpoint in Loki — distributor to ingester, querier to ingester, frontend to querier — enforces grpc_server_max_recv_msg_size and grpc_server_max_send_msg_size (default 4MB). If a push batch, a query result set, or an internal RPC serializes larger than that, gRPC refuses it before the application even sees it. The numbers in the message (X vs. Y) are the actual size versus the configured limit.

Symptoms

• Push or query requests fail with ResourceExhausted and message larger than max and a size pair like (5242880 vs. 4194304).
• Large log batches from Promtail/Alloy fail while small batches succeed.
• Queries returning huge result sets error out; narrower queries return fine.
• The error appears between internal components (distributor↔ingester, frontend↔querier), not just at the edge.
• The reported max is 4194304 (4MB) — the gRPC default — pointing straight at the size limit.

Common Root Causes

• Push batches too large — batchsize in the client exceeds the server’s receive limit.
• Default 4MB gRPC limit left in place while workloads grew.
• Queries with enormous result payloads — very wide ranges or limit values returning more than 4MB.
• Mismatched client/server limits — server raised but client send limit not, or vice versa.
• Large chunk_target_size/chunk transfers during ingester handoff exceeding the RPC cap.

Diagnostic Workflow

Read the exact size pair from the error to know how much headroom you need. Then check the gRPC size limits in the Loki server config:

server:
  grpc_server_max_recv_msg_size: 16777216   # 16MB (default 4194304)
  grpc_server_max_send_msg_size: 16777216   # 16MB
  grpc_server_max_concurrent_streams: 1000

Client-side gRPC limits also matter for internal calls:

# querier/frontend gRPC client to ingesters
ingester_client:
  grpc_client_config:
    max_recv_msg_size: 16777216
    max_send_msg_size: 16777216

frontend_worker:
  grpc_client_config:
    max_send_msg_size: 16777216

For a push-side failure, inspect the shipping agent’s batch size:

# promtail client — shrink the batch so it serializes under the server cap
clients:
  - url: http://loki:3100/loki/api/v1/push
    batchsize: 1048576     # 1MB
    batchwait: 1s

Confirm whether the failing call is a push (edge) or an internal RPC by matching the log source component; that tells you which pair of limits to raise.

Example Root Cause Analysis

A team shipped logs with Alloy using a 5MB batch to reduce request overhead. The distributor received each push as one gRPC message, but grpc_server_max_recv_msg_size was left at the 4MB default. Every batch failed with received message larger than max (5242880 vs. 4194304) and Alloy retried endlessly, so logs backed up and eventually dropped.

They fixed it on both sides. On the server they raised grpc_server_max_recv_msg_size and grpc_server_max_send_msg_size to 16MB to give real headroom. On the client they lowered batchsize to 1MB so a single frame never approached the cap even under bursty logging. They also raised the matching ingester_client.grpc_client_config limits because a wide dashboard query had begun tripping the same error on the querier→ingester path. After the change, both push and query paths worked and the retry storm cleared.

Prevention Best Practices

• Set grpc_server_max_recv_msg_size and grpc_server_max_send_msg_size consistently across all components, with real headroom (16MB is common).
• Match client gRPC limits (ingester_client, frontend_worker) to the server values so no path is the weak link.
• Keep shipper batchsize well under the server receive cap (1MB is safe against a 16MB cap).
• Cap query result payloads with limit and max_entries_limit_per_query so result sets stay bounded.
• After raising limits, watch ingester memory — bigger frames mean bigger buffers.
• Roll config changes to every replica; a single un-updated ingester still rejects large frames.

Quick Command Reference

# Confirm the effective gRPC size limits
curl -s http://loki:3100/config | grep -E 'grpc_server_max_(recv|send)_msg_size'

# Watch for ResourceExhausted in component logs
journalctl -u loki --since '15 min ago' | grep -i 'larger than max'

server:
  grpc_server_max_recv_msg_size: 16777216
  grpc_server_max_send_msg_size: 16777216
ingester_client:
  grpc_client_config:
    max_recv_msg_size: 16777216
    max_send_msg_size: 16777216

Conclusion

ResourceExhausted: grpc: received message larger than max means a gRPC frame between Loki components exceeded the size cap — the message even prints the actual size versus the 4MB default. Raise grpc_server_max_recv_msg_size/grpc_server_max_send_msg_size and the matching client limits consistently across every component, and shrink shipper batch sizes so frames stay well under the cap. Bounding query result sizes and watching ingester memory keeps the larger limits safe in production.