🚨 Prometheus Error Guide: 'load too many samples into memory' — Query Fix

Overview

Prometheus aborts a query with this error when executing it would pull more samples into memory at once than the --query.max-samples limit allows:

query processing would load too many samples into memory in query execution

It is returned by the API and shown in the UI and Grafana panels as a failed query. Unlike a timeout, this is a hard guardrail: Prometheus refuses to run the query rather than risk an OOM. The limit protects the server from a single expensive query taking down monitoring for everyone. The fix is almost always to make the query cheaper, not to raise the ceiling.

Symptoms

• API/UI returns query processing would load too many samples into memory in query execution.
• Heavy dashboards or wide range queries fail while narrow ad-hoc queries succeed.
• Failures scale with time range: a 1h view works, 30d fails.
• Queries touching high-cardinality metrics (no label filter, big group by) trigger it.
• prometheus_engine_queries and query-samples metrics spike around the failure.

Common Root Causes

• Unbounded selectors — a metric with millions of series selected with no label matchers.
• Long ranges at fine resolution — a large [Nd] range or wide query_range with a tiny step multiplies series × points.
• Expensive aggregations over raw data — sum(rate(...)) across the whole fleet computed ad hoc instead of via recording rules.
• --query.max-samples set low for the workload (default is 50,000,000).
• Cardinality explosion — a label (pod, uuid, path) blew up series count, so even a normal query loads too many samples.
• Subqueries with fine resolution over long ranges compounding sample counts.

Diagnostic Workflow

Estimate the query’s cost: how many series it selects and over how many points. Start with the series count of the raw selector:

count({__name__="http_requests_total"})
count by (__name__)({job="api"})

Reproduce against the API and read the error precisely:

curl -sS 'http://localhost:9090/api/v1/query_range' \
  --data-urlencode 'query=sum(rate(http_requests_total[5m]))' \
  --data-urlencode 'start=...' --data-urlencode 'end=...' \
  --data-urlencode 'step=15s' | jq '.error, .errorType'

Check the configured ceiling and current pressure:

ps aux | grep -o '\-\-query.max-samples=[0-9]*'

# Find the biggest cardinality offenders driving cost
topk(10, count by (__name__)({__name__=~".+"}))

Offload heavy repeated aggregations into a recording rule so dashboards read a small pre-aggregated series instead of raw data:

groups:
  - name: api-aggregations
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

Example Root Cause Analysis

A “fleet overview” Grafana dashboard failed on the 7-day view with query processing would load too many samples into memory. The panel ran sum(rate(http_requests_total[5m])) across ~2.1M series at a 15s step over 7 days — series × points far exceeded the 50M max-samples ceiling.

Rather than raise --query.max-samples (which just moves the OOM risk), the metric was pre-aggregated with a recording rule job:http_requests:rate5m evaluated every 30s. The panel switched to querying the recording rule, collapsing millions of raw series into a handful per job. The 7-day view rendered instantly, and the underlying cardinality was separately reduced via metric_relabel_configs.

Prevention Best Practices

• Pre-aggregate expensive, frequently viewed queries into recording rules so dashboards never scan raw high-cardinality data.
• Always constrain selectors with label matchers; avoid bare metric names across the whole fleet.
• Match query_range step to the range so long views downsample instead of loading fine-grained points.
• Keep cardinality in check with metric_relabel_configs and cardinality alerts; runaway labels make every query expensive.
• Treat --query.max-samples as a safety limit, not a tuning knob — raising it hides cost and risks server OOM.

Quick Command Reference

# Reproduce and read the error type
curl -sS 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=YOUR_QUERY' | jq '.error, .errorType'

# Current ceiling
ps aux | grep -o '\-\-query.max-samples=[0-9]*'

count({__name__="metric_name"})
topk(10, count by (__name__)({__name__=~".+"}))

Conclusion

“load too many samples into memory” is Prometheus protecting itself from a query that would scan more than --query.max-samples. The right response is to shrink the query — narrower selectors, shorter ranges, coarser steps, and recording rules for repeated aggregations — plus keeping cardinality under control. Raising the limit only defers an eventual OOM.