🚨 Loki Error Guide: 'query too large to execute on a single querier' — Narrow the Scan Before You Raise the Limit

Overview

Loki’s querier and query frontend estimate how many bytes a query will scan before running it. When that estimate exceeds the configured budget, Loki refuses to run the query and returns an HTTP 400 with a message like this:

query too large to execute on a single querier: (query) would read too many bytes (limit 21474836480 bytes) - consider adding more specific stream selectors, reducing the time range, or bumping max_query_bytes_read

Two limits govern this: limits_config.max_query_bytes_read, which caps the bytes a single sub-query may scan, and limits_config.max_querier_bytes_read, which caps the total across all sub-queries a single querier handles. The error is a guardrail against a broad, unbounded query pulling gigabytes of chunks into memory and starving the read path. The message itself names the three real fixes in priority order: narrow the selector, shrink the range, or (last resort) raise the limit. The right response is almost always to make the query more specific, not to bump the ceiling.

Symptoms

• Queries return HTTP 400 with would read too many bytes (limit ... bytes) instead of results.
• Broad queries like {namespace="prod"} over a wide time range fail while narrow ones succeed.
• Grafana panels with loose selectors error out, especially at longer dashboard time ranges.
• logcli reports the same limit message for exploratory queries with few or no label matchers.
• Querier memory and CPU spike just before the frontend starts rejecting these queries.

Common Root Causes

• Broad stream selectors — {namespace="prod"} with no app/level matcher forces a scan across every stream in the namespace.
• Wide time ranges — a multi-day or multi-week range multiplies the chunk volume scanned by any given selector.
• No label matchers at all — a query that leans entirely on a line filter (|= "error") with a near-empty selector, so Loki must read everything then filter.
• Sharding disabled or ineffective — query sharding is off, or the schema is not TSDB, so the work cannot be split into cheaper parallel sub-queries.
• High-volume streams — a single chatty app produces so many bytes that even a modest range crosses the budget.
• Missing metric/recording rules — dashboards recompute heavy aggregations over raw logs on every load instead of reading pre-aggregated series.

How to diagnose

1. Read the effective byte limits for the tenant so you know the real budget:

   limits_config:
     max_query_bytes_read: 21474836480      # 20GB per sub-query
     max_querier_bytes_read: 107374182400   # 100GB per querier total

2. Measure what the query actually scans with logcli --stats, which reports bytes processed and chunks touched:

   logcli query '{namespace="prod"} |= "error"' \
     --stats --limit=10 --since=24h \
     --addr=https://loki-gateway/ --org-id=tenant-a

3. Confirm the schema uses TSDB so query sharding can actually split the work:

   schema_config:
     configs:
       - from: 2024-01-01
         store: tsdb
         object_store: s3
         schema: v13
         index:
           prefix: index_
           period: 24h

4. Verify query sharding is enabled on the frontend/querier path:

   limits_config:
     tsdb_max_query_parallelism: 128
     split_queries_by_interval: 30m

5. Identify the heaviest streams so you know which selector to tighten:

   logcli series '{namespace="prod"}' --analyze-labels \
     --addr=https://loki-gateway/ --org-id=tenant-a

Fixes

Add specific label matchers so Loki scans only the streams you care about instead of the whole namespace:

{namespace="prod", app="checkout", level="error"} |= "timeout"

Shorten the time range — split a broad investigation into narrower windows, which cuts the bytes scanned proportionally:

logcli query '{namespace="prod", app="checkout"}' \
  --since=1h --limit=100 \
  --addr=https://loki-gateway/ --org-id=tenant-a

Enable TSDB query sharding and interval splitting so the query is divided into cheaper parallel sub-queries, each well under the per-sub-query budget:

limits_config:
  split_queries_by_interval: 15m
  tsdb_max_query_parallelism: 256

Precompute heavy aggregations with recording rules so dashboards read a small metric series instead of rescanning raw logs each load:

groups:
  - name: loki-recording
    rules:
      - record: job:checkout_errors:rate5m
        expr: sum(rate({app="checkout", level="error"}[5m]))

Raise the byte limit only after narrowing has failed — bump max_query_bytes_read deliberately for a tenant that genuinely needs larger scans, and size the queriers to match:

limits_config:
  max_query_bytes_read: 42949672960   # 40GB; raise cautiously, watch memory

What to watch out for

• Raising max_query_bytes_read lets the query run but shifts the cost onto querier memory and latency — a big enough scan can OOM a querier and take out unrelated queries.
• Sharding only helps if the schema is TSDB and parallelism is set; on a legacy boltdb-shipper schema you cannot split the work the same way.
• max_query_bytes_read (per sub-query) and max_querier_bytes_read (per querier total) are different knobs — a query can pass one and fail the other.
• Line filters do not reduce bytes scanned; only label matchers and a tighter range do. Adding |= "error" to a broad selector still reads everything first.
• Always confirm with --stats after a change; a fix that looks tighter but still scans the same chunks has not actually helped.

Related

• Loki Error Guide: ‘the query time range exceeds the limit’ — the range-length guardrail that pairs with the byte budget.
• Loki Error Guide: ‘too many chunks to fetch’ — the chunk-count sibling of scanning too many bytes.
• Loki Error Guide: ‘maximum of series reached’ — the cardinality limit that broad selectors also trip.