🚨 VictoriaMetrics Error Guide: 'cannot execute query: context deadline exceeded' — Raise Timeout or Optimize the Query

Overview

VictoriaMetrics returns this error when a query runs longer than the configured maximum query duration, or when the underlying storage layer cannot return data before the request context is cancelled. It is surfaced in vmui, in Grafana panels, and in the raw HTTP API response:

cannot execute query: context deadline exceeded

On a cluster setup the same condition often appears with a storage-side qualifier when vmselect gives up waiting on vmstorage:

cannot execute query on all the vmstorage nodes: context deadline exceeded

The deadline is governed primarily by -search.maxQueryDuration (default 30s). When a query’s execution time crosses that budget, VictoriaMetrics cancels the request context and every downstream operation aborts with context deadline exceeded. It is a symptom of one of three things: a genuinely expensive query, an undersized or overloaded vmstorage, or a timeout that is simply set too low for the workload.

Symptoms

• Grafana panels show context deadline exceeded instead of data, usually the wide-range or high-cardinality panels.
• The same query succeeds over a short time range (last 1h) but fails over 7d or 30d.
• vmui’s query trace stops partway with a cancelled context.
• Errors cluster around dashboard refreshes or recording-rule evaluation, when many queries land at once.
• vmselect logs show cannot execute query on all the vmstorage nodes and CPU on vmstorage spikes to saturation.
• The failing query touches many time series (label matchers that select thousands of series, or {__name__=~".+"}).

Common Root Causes

• Query exceeds -search.maxQueryDuration — a heavy aggregation over a long range simply cannot finish in the default 30s budget.
• High-cardinality selectors — matchers that expand to tens of thousands of series force vmstorage to scan and merge enormous amounts of data.
• Slow or overloaded vmstorage — disk IO saturation, CPU starvation, or a compaction storm makes reads slow enough to blow the deadline.
• Undersized cluster — too few vmstorage nodes for the ingested volume, so each query fans out to overloaded shards.
• Expensive functions — subqueries, rate() over large ranges, topk() across huge series counts, or regex matchers on unindexed labels.
• Network latency between vmselect and vmstorage — cross-AZ or cross-region hops eat into the deadline before any data is read.
• Client-side timeout shorter than the server’s — Grafana’s own datasource timeout may cancel before VictoriaMetrics does, producing the same class of error.

Diagnostic Workflow

First confirm the configured limits on the node handling the query. On single-node VictoriaMetrics (port 8428), check the runtime flags:

curl -s http://<node>:8428/flags | grep -i 'maxQueryDuration\|maxPointsPerTimeseries\|maxUniqueTimeseries'

On a cluster, the deadline is enforced by vmselect (port 8481):

curl -s http://<vmselect>:8481/flags | grep -i 'maxQueryDuration\|maxConcurrentRequests'

Reproduce the failing query against the API and time it, so you know how far over budget you are:

time curl -s 'http://<node>:8428/api/v1/query_range' \
  --data-urlencode 'query=sum(rate(http_requests_total[5m])) by (pod)' \
  --data-urlencode 'start=2026-07-01T00:00:00Z' \
  --data-urlencode 'end=2026-07-08T00:00:00Z' \
  --data-urlencode 'step=60s'

Use vmui’s built-in query tracing (the “Enable query tracing” toggle) to see which phase dominates — index lookup, data read, or merge. In vmui at http://<node>:8428/vmui/ the trace tree shows per-node timings on a cluster.

Measure how many series the selector actually matches, since cardinality is the usual culprit:

curl -s 'http://<node>:8428/api/v1/series/count'
curl -s 'http://<node>:8428/api/v1/status/tsdb' | head -40

Inspect the storage layer for saturation. Scrape vmstorage metrics and look at slow-read and disk indicators:

curl -s http://<vmstorage>:8482/metrics | grep -E 'vm_slow_queries_total|vm_rows_read_per_query|process_resident_memory_bytes'
curl -s http://<vmstorage>:8482/metrics | grep -E 'vm_free_disk_space_bytes|vm_data_size_bytes'

Check the service logs for the deadline and any storage-node errors:

journalctl -u victoriametrics --since '30 min ago' | grep -i 'deadline\|cannot execute'
journalctl -u vmselect --since '30 min ago' | grep -i 'deadline\|vmstorage'

If the cluster is involved, verify vmselect can reach every vmstorage on the query port 8401:

for n in vmstorage-0 vmstorage-1 vmstorage-2; do nc -zv $n 8401; done

Example Root Cause Analysis

A team paged on Grafana dashboards showing context deadline exceeded every morning at 09:00. Short-range panels rendered fine; the 30-day capacity panels failed.

Running the failing panel query with time showed it took 41s before the server cancelled it — over the default -search.maxQueryDuration=30s. vmui’s query trace revealed that 90% of the time was spent in the data-read phase on vmstorage, not in index lookup. Scraping vm_rows_read_per_query on vmstorage:8482 showed the panel touched ~180M rows because the selector was sum(rate(node_network_receive_bytes_total[5m])) with no by/without grouping and no namespace filter, expanding to every interface on every node across the fleet.

The 09:00 clustering was the giveaway: a nightly recording-rule batch plus the morning dashboard load hit vmstorage simultaneously, and disk read IO saturated. process_resident_memory_bytes on vmstorage was near the -memory.allowedPercent ceiling, forcing evictions and re-reads.

The fix was layered: the panel query was scoped to a single namespace and pre-aggregated with a recording rule, cutting rows read by 95%; -search.maxQueryDuration was raised to 90s on vmselect to give legitimate long-range queries headroom; and a fourth vmstorage node was added to spread the read load. The error stopped recurring.

Prevention Best Practices

• Pre-aggregate expensive dashboard queries with recording rules (vmalert) so panels read a small, low-cardinality series instead of computing rate() over millions of rows on every refresh.
• Set -search.maxQueryDuration to match real workload needs (e.g. 60s–120s), but treat frequent timeouts as a signal to fix the query, not just raise the limit.
• Bound cardinality: enforce label discipline, avoid unbounded labels (request IDs, full URLs), and watch vm_rows growth and /api/v1/status/tsdb for cardinality creep.
• Scale vmstorage horizontally when read latency grows; more nodes mean each shard scans fewer rows per query.
• Keep -memory.allowedPercent sane (default 60) and give vmstorage enough RAM to cache index and recent data, avoiding disk re-reads.
• Align client timeouts (Grafana datasource timeout) with the server’s -search.maxQueryDuration so you get a clear server-side error rather than an ambiguous client cancel.
• Alert on vm_slow_queries_total increasing so you catch expensive queries before they page.

Quick Command Reference

# Show the active query-duration limit (single-node 8428 / cluster vmselect 8481)
curl -s http://<node>:8428/flags | grep -i maxQueryDuration
curl -s http://<vmselect>:8481/flags | grep -i maxQueryDuration

# Time the failing query directly against the API
time curl -s 'http://<node>:8428/api/v1/query_range' \
  --data-urlencode 'query=<your MetricsQL>' \
  --data-urlencode 'start=...' --data-urlencode 'end=...' --data-urlencode 'step=60s'

# Cardinality / TSDB status
curl -s 'http://<node>:8428/api/v1/status/tsdb' | head -40

# Storage saturation signals
curl -s http://<vmstorage>:8482/metrics | grep -E 'vm_slow_queries_total|vm_rows_read_per_query'

# Verify vmselect -> vmstorage query port
nc -zv <vmstorage> 8401

# Logs
journalctl -u victoriametrics --since '30 min ago' | grep -i deadline

# Raise the limit (flag on single-node or vmselect)
-search.maxQueryDuration=90s

Conclusion

cannot execute query: context deadline exceeded means VictoriaMetrics could not finish a query inside its allotted time. The durable fix is rarely to just raise -search.maxQueryDuration — start by measuring how far over budget the query runs, use vmui tracing to find whether index, read, or merge dominates, and check vmstorage for saturation. Pre-aggregate heavy queries with recording rules, scope selectors to cut cardinality, and scale storage when reads are genuinely slow. Reserve the timeout increase for legitimately long-range queries once the query itself is efficient.