🚨 Loki Error Guide: 'maximum of series reached for a single query' — Reduce Cardinality or Raise the Limit

Overview

Loki aborts a metric query when the number of distinct series it would return exceeds max_query_series. Grafana or logcli shows:

maximum of series (500) reached for a single query

This is a read-path guardrail, not an ingestion problem. It fires when a LogQL metric query (anything with rate, count_over_time, sum by (...), etc.) produces more result series than the limit allows — usually because the by (...) grouping includes a high-cardinality label.

Symptoms

• A LogQL metric query returns the error instead of data; log-only queries (no aggregation) still work.
• The query works over a short range but fails over a longer one as more series appear.
• Adding a high-cardinality label to by (...) (e.g. pod, instance, request_id) triggers it immediately.
• Grafana panels using template variables that expand to many series break.
• Querier logs show maximum of series ... reached for a single query.

Common Root Causes

• High-cardinality grouping — sum by (pod) or by (instance) over a large fleet produces thousands of series.
• Unbounded label matchers — a query with only a broad matcher like {cluster="prod"} spans every stream in the cluster.
• Parsed labels in grouping — grouping by a field extracted with | json/| logfmt that has many distinct values.
• Wide time ranges — longer ranges surface more historically active streams.
• Limit too low for legitimate use — max_query_series left at a conservative default for a large but valid dashboard.
• Accidental fan-out — a missing aggregation so each stream stays its own series.

Diagnostic Workflow

First, see how many series the matcher covers before aggregating:

count(count by (pod, namespace, app) (rate({cluster="prod"}[5m])))

Find which label is exploding cardinality:

logcli series '{cluster="prod"}' --since=1h --analyze-labels

--analyze-labels prints each label and its distinct value count — the offender is the label with thousands of values.

Check the configured limit:

limits_config:
  max_query_series: 500

Rewrite the query to aggregate away the high-cardinality dimension. Instead of:

sum by (pod) (rate({namespace="payments"} |= "error" [5m]))

group by a low-cardinality dimension:

sum by (namespace, level) (rate({namespace="payments"} |= "error" [5m]))

Or keep the detail but bound the result with topk:

topk(20, sum by (pod) (rate({namespace="payments"} |= "error" [5m])))

Example Root Cause Analysis

An SRE built an error-rate panel with sum by (pod) (rate({namespace="payments"} |= "error" [5m])). It worked in staging (12 pods) but failed in production with maximum of series (500) reached. Running logcli series '{namespace="payments"}' --analyze-labels showed pod had 1,800 distinct values across the query window because of frequent rollouts and HPA scaling — every pod name, including terminated ones, counted as a series.

The fix was to group by the stable deployment label instead of pod, collapsing 1,800 series into 6, and to add topk(25, ...) on a secondary drill-down panel where per-pod detail was genuinely wanted. The panel rendered instantly and the limit was left untouched — the problem was query cardinality, not an undersized limit. Raising max_query_series would have masked a query that scanned far more series than the dashboard needed.

Prevention Best Practices

• Group metric queries by low-cardinality labels (namespace, app, deployment, level) and avoid pod/instance/request_id in by (...).
• Use topk/bottomk to bound result series when per-instance detail is needed.
• Run logcli series --analyze-labels when authoring dashboards to see cardinality before shipping.
• Raise max_query_series only for specific tenants with legitimately large-but-bounded result sets, and watch querier memory when you do.
• Keep high-cardinality fields out of stream labels entirely (use structured metadata) so they can’t accidentally be grouped on cheaply.
• Prefer narrower label matchers over broad {cluster="..."} selects.

Quick Command Reference

# Count series a matcher would return
logcli query 'count(count by (pod) (rate({cluster="prod"}[5m])))' --since=1h

# Analyze label cardinality for a selector
logcli series '{namespace="payments"}' --since=1h --analyze-labels

# Bound a high-cardinality query
logcli query 'topk(20, sum by (pod) (rate({namespace="payments"} |= "error"[5m])))' --since=1h

# Inspect the configured limit
logcli --addr=http://loki-gateway/ config | grep -i max_query_series

Conclusion

maximum of series reached for a single query means your LogQL metric query would materialize more series than max_query_series permits — almost always because a high-cardinality label sits in the by (...) clause. Use --analyze-labels to find the offender, then aggregate by a stable low-cardinality dimension or bound the result with topk. Reserve raising the limit for genuinely large, bounded dashboards, and keep high-cardinality fields out of labels so this guardrail rarely trips.