🚨 Loki Error Guide: 'too many outstanding requests' — Raise Querier Concurrency and Split Queries

Overview

Grafana Loki returns this error when the query frontend’s per-tenant queue is full and it cannot accept another sub-query. Grafana or logcli surfaces it as an HTTP 429:

too many outstanding requests

Loki splits every query into many sub-queries by time and by shard, then places them on an in-memory queue that queriers pull from. max_outstanding_per_tenant bounds the depth of that queue for each tenant. When queriers cannot drain sub-queries as fast as the frontend enqueues them, the queue hits its cap and Loki rejects further sub-queries with too many outstanding requests. It is a saturation signal, not a syntax error: the read path is under-provisioned relative to the query load.

Symptoms

• Dashboards fail intermittently with HTTP 429 and the message too many outstanding requests, often only for wide time ranges.
• logcli query fails immediately on large ranges but succeeds on a 15-minute window.
• The error appears in bursts when several users open heavy dashboards at once.
• cortex_query_frontend_queue_length climbs and stays high in the query-frontend metrics.
• Small queries still work, so Loki is clearly up — only large or concurrent queries are rejected.

Common Root Causes

• Too few queriers or too little querier concurrency — the queue fills faster than it drains.
• Low max_outstanding_per_tenant — the default (100 in older versions, 2048 in newer) is too small for the query volume.
• Very large time ranges with fine split_queries_by_interval — a 30-day query split into 30-minute chunks becomes thousands of sub-queries.
• Aggressive query sharding (TSDB with tsdb_max_query_parallelism) multiplying sub-query count.
• Slow object storage making each sub-query take long, so queriers drain slowly.
• No query-scheduler in large deployments, so the frontend queue is the only buffer.

Diagnostic Workflow

First confirm it is a 429 and read the exact body with logcli in verbose mode:

logcli --stats query '{app="checkout"}' --since=720h
# HTTP 429 too many outstanding requests

Inspect the frontend and querier limits actually in effect:

# loki config: relevant read-path knobs
query_range:
  split_queries_by_interval: 30m   # smaller = MORE sub-queries
  parallelise_shardable_queries: true

frontend:
  max_outstanding_per_tenant: 2048  # queue depth per tenant

querier:
  max_concurrent: 8                 # sub-queries a single querier runs at once

limits_config:
  tsdb_max_query_parallelism: 128
  max_query_parallelism: 32

Check queue depth and querier saturation from the metrics endpoint:

# On the query-frontend / scheduler
curl -s http://loki-query-frontend:3100/metrics | grep -E 'query_frontend_queue_length|cortex_query_scheduler_queue_length'
# On a querier
curl -s http://loki-querier:3100/metrics | grep -E 'loki_querier_.*inflight|cortex_querier_'

Compare enqueue rate against querier count. If queue_length is consistently near max_outstanding_per_tenant, the read path is under-scaled, not misconfigured.

Example Root Cause Analysis

A team ran Loki in microservices mode with three queriers, each at max_concurrent: 4, giving 12 concurrent sub-query slots. split_queries_by_interval was set to 15m. An SRE opened a dashboard covering the last 14 days: 14d ÷ 15m = 1344 time splits, multiplied by shard parallelism, producing several thousand sub-queries instantly. With only 12 drain slots and max_outstanding_per_tenant: 100, the queue saturated in under a second and every panel returned too many outstanding requests.

The fix was three-pronged: raised max_outstanding_per_tenant to 2048, raised querier.max_concurrent to 8 and scaled queriers from 3 to 6 (48 drain slots), and widened split_queries_by_interval to 30m to halve the sub-query count. After the change the same 14-day dashboard loaded in four seconds with the queue never exceeding 400.

Prevention Best Practices

• Scale queriers and querier.max_concurrent together so total drain capacity matches peak sub-query enqueue rate.
• Keep max_outstanding_per_tenant at 2048 or higher for busy tenants; it is cheap in-memory queue headroom.
• Deploy the query-scheduler component in large clusters so the frontend is decoupled from queue depth.
• Tune split_queries_by_interval so typical dashboard ranges produce hundreds, not thousands, of splits.
• Add label filters early so shardable queries touch fewer streams.
• Alert on cortex_query_frontend_queue_length approaching the tenant cap before users see 429s.

Quick Command Reference

# Reproduce with a wide range
logcli --stats query '{namespace="prod"}' --since=720h

# Watch queue depth on the frontend/scheduler
watch -n2 "curl -s localhost:3100/metrics | grep queue_length"

# Confirm running config values
curl -s http://loki:3100/config | grep -A3 -E 'max_outstanding|split_queries|max_concurrent'

# Balanced read-path settings
frontend:
  max_outstanding_per_tenant: 2048
querier:
  max_concurrent: 8
query_range:
  split_queries_by_interval: 30m
  parallelise_shardable_queries: true

Conclusion

too many outstanding requests is Loki telling you the read path cannot keep up: the per-tenant sub-query queue is full. Treat it as a capacity problem first — add querier concurrency and instances, raise max_outstanding_per_tenant, and reduce sub-query explosion by widening the split interval. Deploying a query-scheduler and alerting on queue length keeps the queue from ever reaching its cap during peak dashboard load.