🚨 Loki Error Guide: 'context deadline exceeded' — Fix Query Timeouts on the Read Path

Overview

Loki returns this Go error when a query or an internal read operation runs longer than its allotted deadline and the context is cancelled:

rpc error: code = Code(504) desc = context deadline exceeded

The plain gRPC form also appears in querier and ingester logs:

context deadline exceeded

Every Loki query carries a deadline derived from querier.query_timeout and the HTTP server write timeout. When a sub-query — reading chunks from object storage, scanning an index, or waiting on an ingester — does not finish before that deadline, Go cancels the context and the operation fails. It means the work took too long, not that the query was wrong. The cause is almost always an oversized time range, missing label matchers, or slow storage.

Symptoms

• Wide-range dashboard panels fail with a 504 and context deadline exceeded, while short ranges load fine.
• logcli query over several days times out at a suspiciously round number of seconds (30s, 60s, 300s).
• Ingester or querier logs show context deadline exceeded around chunk fetches or Store.SelectLogs.
• The failure time matches a configured timeout exactly, indicating a deadline rather than a crash.
• Adding a tight label filter or shrinking the range makes the same query succeed.

Common Root Causes

• querier.query_timeout too low for the query’s honest workload.
• HTTP timeouts shorter than the query timeout — http_server_write_timeout cancels the connection before Loki finishes.
• Broad queries with few or no label matchers forcing full scans of many streams.
• Slow or throttled object storage (S3/GCS) so chunk fetches stall.
• Under-provisioned queriers so sub-queries wait in queue and burn the deadline.
• Reverse proxy timeouts (nginx/Envoy) shorter than Loki’s own, cancelling upstream.

Diagnostic Workflow

Reproduce with --stats so you can see where time is spent:

logcli --stats query '{cluster="prod"}' --since=168h
# Ingester.TotalReached, Store.TotalChunksDownloaded, Summary.ExecTime

Read the relevant timeout keys from config — they must be ordered outer >= inner:

server:
  http_server_read_timeout: 310s
  http_server_write_timeout: 310s   # must exceed query timeout

querier:
  query_timeout: 300s
  engine:
    timeout: 300s

limits_config:
  query_timeout: 300s               # per-tenant override (newer versions)

Confirm no proxy is cancelling first:

# nginx in front of Loki must allow the longer window
grep -R 'proxy_read_timeout\|proxy_send_timeout' /etc/nginx/
# Envoy route timeout must be >= Loki query_timeout

Isolate slow storage by timing a narrow high-cardinality query versus a wide one; if Store.TotalChunksDownloaded is huge, the bottleneck is storage read volume, not the deadline itself.

Example Root Cause Analysis

A platform team hit context deadline exceeded on any dashboard wider than a few hours. Their querier.query_timeout was 300s, but nginx sat in front with a default proxy_read_timeout of 60s. Loki was happily working, yet nginx cancelled the upstream connection at exactly 60 seconds and returned 504 — which Loki logged as context deadline exceeded because the client context was gone.

They also had queries like {env="prod"} |= "error" with no service label, scanning thousands of streams. The fix: set nginx proxy_read_timeout 310s to sit just above Loki’s 300s, and educate users to always include a service or app matcher. Chunk download volume dropped 90 percent and 504s disappeared; the few genuinely heavy queries finished in 40 seconds, well inside the deadline.

Prevention Best Practices

• Order timeouts consistently: proxy timeout >= http_server_write_timeout >= querier.query_timeout.
• Require at least one specific label matcher (namespace, app, service) in dashboards and alerts.
• Use metric queries and recording rules instead of scanning raw logs for repeated aggregations.
• Keep object storage close and fast; watch S3 request latency and retry rates.
• Scale queriers so sub-queries do not sit in queue consuming the deadline.
• Set sensible per-tenant limits_config.query_timeout so no single tenant monopolizes read capacity.

Quick Command Reference

# See execution time and chunk volume
logcli --stats query '{app="api"}' --since=24h

# Check effective timeouts in the running config
curl -s http://loki:3100/config | grep -iE 'query_timeout|write_timeout|read_timeout'

# Confirm proxy is not the limiter
grep -R proxy_read_timeout /etc/nginx/

# Coherent timeout stack
server:
  http_server_write_timeout: 310s
querier:
  query_timeout: 300s

Conclusion

context deadline exceeded means a Loki read exceeded its deadline and got cancelled. Fix it by making timeouts consistent from the proxy down to querier.query_timeout, then attack the real slowness: narrow ranges, add label matchers, speed up object storage, and scale queriers. Raising the timeout alone only hides an under-provisioned read path — pair it with query hygiene so genuine queries finish comfortably inside the deadline.