🚨 Grafana Error Guide: 'maximum of series reached' — fixing Loki query limits in Grafana

Overview

When you run a high-cardinality LogQL query from a Grafana panel against a Loki datasource, Loki enforces server-side limits and rejects the query rather than melting the backend. Grafana surfaces the rejection verbatim in the panel and in the query inspector.

maximum of series (500) reached for a single query
max entries limit per query exceeded, limit > max_entries_limit
too many outstanding requests
rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max
the query would read too many chunks

These are guardrails in Loki’s limits_config, not bugs. maximum of series (N) reached means your query matches more log streams than max_query_series allows; too many outstanding requests means the query-frontend/scheduler queue for your tenant is saturated. The fix is almost always to make the query more selective, and secondarily to tune the limits.

Symptoms

• A Loki panel shows red with maximum of series (500) reached for a single query.
• Broad queries like {job="app"} fail while narrow ones succeed.
• Dashboards with many concurrent Loki panels intermittently return too many outstanding requests.
• Long time ranges trigger the query would read too many chunks or ResourceExhausted.
• The Grafana query inspector shows a 4xx/5xx from /loki/api/v1/query_range.

Common Root Causes

1. Query matches too many streams (high cardinality)

A selector with no additional filters can match thousands of streams. Loki caps returned series at max_query_series (default 500).

# BAD: matches every stream for the job -> explodes past 500 series
{job="app"}

# GOOD: narrow by label + line filter
{job="app", level="error"} |= "timeout"

maximum of series (500) reached for a single query

High-cardinality labels (e.g. pod, request_id, ip) are the usual culprit — each unique value is a new stream.

2. Result set exceeds max entries per query

Returning too many raw log lines trips max_entries_limit_per_query (default 5000).

# loki config: limits_config
limits_config:
  max_entries_limit_per_query: 5000

max entries limit per query exceeded, limit > max_entries_limit

On the Grafana side, the datasource Maximum lines (maxLines, default 1000) also caps rows requested.

3. Query-scheduler / frontend saturation

too many outstanding requests means the per-tenant queue in the query-scheduler is full — too many splits or too many concurrent dashboard panels.

query_scheduler:
  max_outstanding_requests_per_tenant: 2048
limits_config:
  split_queries_by_interval: 30m   # fewer splits = fewer queued subqueries
  max_query_parallelism: 32
  tsdb_max_query_parallelism: 512

too many outstanding requests

4. Time range too large / too many chunks

A 30-day range over a busy stream reads enormous chunk volumes.

the query would read too many chunks
rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (...)

Diagnostic Workflow

Step 1 — Reproduce the query against Loki directly to see the raw limit.

curl -s -G "http://loki:3100/loki/api/v1/query_range" \
  --data-urlencode 'query={job="app"}' \
  --data-urlencode 'start='$(date -d '1 hour ago' +%s)'000000000' \
  --data-urlencode 'end='$(date +%s)'000000000' \
  --data-urlencode 'limit=1000' | jq '.status, .error'

Step 2 — Count how many streams the selector matches.

# In Grafana Explore, run as a metric query to see stream count
count(count by (pod, level) (rate({job="app"}[5m])))

Step 3 — Inspect current Loki limits.

curl -s http://loki:3100/config | grep -iA2 "max_query_series\|max_entries_limit\|split_queries"
kubectl logs deploy/loki -n logging | grep -iE "maximum of series|outstanding requests"

Step 4 — Narrow the query. Add stream selectors and a line filter:

{job="app", namespace="checkout", level="error"} |= "timeout" | json | status="500"

Step 5 — If limits are legitimately too low, raise them in limits_config.

limits_config:
  max_query_series: 1000
  max_entries_limit_per_query: 10000
  split_queries_by_interval: 30m
query_scheduler:
  max_outstanding_requests_per_tenant: 4096

Example Root Cause Analysis

An engineer built a “All app logs” dashboard panel using {job="app"} over the last 24 hours. It failed instantly:

maximum of series (500) reached for a single query

Running the stream-count query in Explore revealed 2,300 unique streams because the app job was labeled with a high-cardinality pod label and every pod restart minted new streams. Raising max_query_series would have masked the real cost, so instead they:

1. Rewrote the panel to {job="app", level="error"} |= "OrderService" — cutting matched streams to 18.
2. Removed pod from the ingestion pipeline’s structured labels and moved it to | json extraction at query time.
3. Set the datasource Maximum lines to 500 for that dashboard.

The panel returned in under a second. They kept max_query_series at 500 as a healthy guardrail rather than papering over the cardinality problem.

Prevention Best Practices

• Always start LogQL with a specific stream selector plus a line filter; never {job="x"} alone on busy jobs.
• Keep labels low cardinality — no pod, request_id, ip, or trace_id as stream labels; extract them at query time with | json/| logfmt.
• Set sensible datasource maxLines per dashboard use case.
• Tune split_queries_by_interval and max_outstanding_requests_per_tenant together to avoid scheduler saturation from many-panel dashboards.
• Use recording rules / metric queries for long ranges instead of streaming raw lines.
• Monitor loki_request_duration_seconds and rejected-query counts to catch abusive queries early.

More Grafana troubleshooting at /categories/grafana/.

Quick Command Reference

# Reproduce a range query directly against Loki
curl -s -G "http://loki:3100/loki/api/v1/query_range" \
  --data-urlencode 'query={job="app", level="error"} |= "timeout"' \
  --data-urlencode 'start='$(date -d '1 hour ago' +%s)'000000000' \
  --data-urlencode 'end='$(date +%s)'000000000' \
  --data-urlencode 'limit=500' | jq '.status, .error'

# Show effective Loki limits
curl -s http://loki:3100/config | grep -iE "max_query_series|max_entries_limit|split_queries|outstanding"

# Grep Loki for limit rejections
kubectl logs deploy/loki -n logging | grep -iE "maximum of series|outstanding requests|too many chunks"

# Count streams a selector matches (run in Grafana Explore)
# count(count by (pod, level) (rate({job="app"}[5m])))

Conclusion

The top root causes of maximum of series reached and related Loki limit errors in Grafana:

1. Too many matched streams — a broad selector exceeds max_query_series (default 500); add label + line filters.
2. Too many returned lines — result trips max_entries_limit_per_query (default 5000) or the datasource maxLines.
3. Scheduler saturation — too many outstanding requests from excessive splits/concurrent panels; tune max_outstanding_requests_per_tenant and split_queries_by_interval.
4. Time range too large — the query would read too many chunks / ResourceExhausted; shorten the range or use recording rules.
5. High-cardinality labels — pod/request_id/ip as stream labels; move them to query-time extraction.