🚨 Grafana Error Guide: 'query processing would load too many samples into memory' — Fix Heavy Panels

Overview

When a Grafana panel runs a PromQL query against Prometheus (or a compatible backend like Thanos, Mimir, or Cortex), the backend has a hard cap on how many samples a single query may load into memory. A broad range, a small step, or a high-cardinality metric can force the engine past that cap, and it aborts the query rather than risk an out-of-memory kill. Grafana surfaces the backend’s error text on the panel.

The literal errors you will see on the panel or in the query response:

query processing would load too many samples into memory in query execution

{"status":"error","errorType":"execution","error":"query processing would load too many samples into memory in query execution"}

# Thanos/Mimir phrasing of the same limit:
expanding series: query processing would load too many samples into memory

This is a protective limit (query.max-samples, default 50,000,000 in Prometheus), not a crash — the backend refused an unbounded query on purpose.

Symptoms

• A specific heavy panel errors; lighter panels on the same datasource are fine.
• The error appears only at wide time ranges (“Last 30 days”) and disappears when you zoom in.
• High-cardinality metrics (per-pod, per-path, per-user) trigger it; low-cardinality ones don’t.
• Increasing the panel’s “Max data points” / decreasing step makes it worse.
• Prometheus logs show the query aborted with the same message.

Common Root Causes

1. Range × resolution × cardinality is simply too big

Samples loaded ≈ (number of series) × (points per series). A rate() over 30 days at a 15s step across thousands of series blows past 50M fast.

2. High-cardinality metric or label

A metric labelled by pod, path, user_id, or container_id can have tens of thousands of series; selecting all of them multiplies the sample count.

3. Sub-query or nested range vector

max_over_time(rate(metric[1m])[7d:15s]) expands into an enormous number of evaluated points.

4. step far smaller than needed

A query_range with a tiny step over a long range computes vastly more points than the panel has pixels to show.

5. No recording rule for an expensive aggregation

Computing histogram_quantile or a big sum by (...) at query time re-reads all raw buckets every render.

Diagnostic Workflow

Step 1: Read the interpolated query and its shape

Open the panel’s Query Inspector (Panel → Inspect → Query) and copy the exact interpolated PromQL, the start/end, and the step.

Step 2: Estimate the series count

# How many series does the selector match? (this is the multiplier)
curl -s -G "http://prometheus:9090/api/v1/series" \
  --data-urlencode 'match[]=http_request_duration_seconds_bucket' \
  | jq '.data | length'

# Which labels blow up cardinality?
curl -s "http://prometheus:9090/api/v1/status/tsdb" \
  | jq '.data.seriesCountByMetricName[0:10], .data.labelValueCountByLabelName[0:10]'

Step 3: Reproduce and time the query directly

time curl -s -G "http://prometheus:9090/api/v1/query_range" \
  --data-urlencode 'query=histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))' \
  --data-urlencode "start=$(date -d '-30 days' +%s)" \
  --data-urlencode "end=$(date +%s)" \
  --data-urlencode 'step=60' | jq '.error // "ok"'

If this returns the “too many samples” error, it’s the query, not Grafana.

Step 4: Make the query cheaper (preferred fix)

Raise the step, reduce cardinality with sum by (...)/without, and shorten the range:

# Aggregate away high-cardinality labels; keep only what the panel shows
sum by (service) (
  rate(http_requests_total{env="$env"}[$__rate_interval])
)

Step 5: Precompute with a recording rule

# prometheus rules file
groups:
  - name: http-slo
    interval: 30s
    rules:
      - record: job:http_request_duration_seconds:p95
        expr: histogram_quantile(0.95, sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))

Then the panel reads a single cheap series: job:http_request_duration_seconds:p95.

Step 6: Tune the backend limit only as a last resort

# Prometheus flag (default 50000000)
--query.max-samples=100000000

# Mimir / Cortex per-tenant limit
limits:
  max_fetched_samples_per_query: 100000000

Raising the cap trades protection for memory pressure — fix the query first.

Example Root Cause Analysis

A latency dashboard’s p95 panel worked at “Last 6 hours” but errored at “Last 30 days” with query processing would load too many samples into memory. The Query Inspector showed histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) over 30 days at a 30s step.

Checking /api/v1/series for the bucket metric returned ~42,000 series — the histogram was labelled by path and pod, so every le bucket multiplied across both. At a 30s step over 30 days (~86,400 points) against tens of thousands of series, the query blew past the 50M sample cap.

Fix: add a recording rule job:http_request_duration_seconds:p95 evaluated every 30s that aggregates away path and pod, and point the panel at it. The panel now reads one precomputed series and loads instantly at any range. Raising --query.max-samples was rejected in review because it would have masked a cardinality problem and risked OOMing Prometheus under concurrent loads. The root cause was cardinality × range, not a backend misconfiguration.

Prevention Best Practices

• Precompute expensive aggregations (percentiles, big sum by) with recording rules so dashboards read cheap series; see more Grafana guides.
• Aggregate away high-cardinality labels in the panel query — show only the dimensions the panel actually displays.
• Set a sane “Max data points” per panel; more points than screen pixels is wasted samples.
• Watch metric cardinality (/api/v1/status/tsdb) and drop unbounded labels like user_id/path at ingest with relabeling.
• Keep --query.max-samples as a guardrail, not a knob to keep raising.
• Triage recurring heavy-query failures with the free monitoring assistant.

Quick Command Reference

# Count series behind the selector (the cardinality multiplier)
curl -s -G "http://prometheus:9090/api/v1/series" \
  --data-urlencode 'match[]=<metric>' | jq '.data | length'

# Cardinality overview
curl -s "http://prometheus:9090/api/v1/status/tsdb" \
  | jq '.data.seriesCountByMetricName[0:10]'

# Reproduce the failing query directly
time curl -s -G "http://prometheus:9090/api/v1/query_range" \
  --data-urlencode 'query=<interpolated query>' \
  --data-urlencode "start=$(date -d '-30 days' +%s)" \
  --data-urlencode "end=$(date +%s)" \
  --data-urlencode 'step=60' | jq '.error // "ok"'

Conclusion

“Query processing would load too many samples into memory” is Prometheus protecting itself from an unbounded query. Fix it in order:

1. Read the interpolated query, range, and step from the Query Inspector.
2. Measure the series count — cardinality is usually the real multiplier.
3. Make the query cheaper: aggregate away labels, raise the step, shorten the range.
4. Precompute with recording rules so dashboards read cheap precomputed series.
5. Raise query.max-samples only as a deliberate, monitored last resort.

Trimming cardinality and adding recording rules removes the cause; raising the sample cap only defers the next OOM.