🚨 VictoriaMetrics Error Guide: 'cannot handle more than -search.maxConcurrentRequests concurrent requests' — Tame Query Load

Overview

VictoriaMetrics limits how many read requests it processes at once. When that limit is reached and a new query has waited past the queue timeout, the server aborts it with HTTP 503 and this message (from single-node victoria-metrics or vmselect in cluster mode):

cannot handle more than 8 concurrent search requests during 10s; possible solutions: to increase `-search.maxQueueDuration`; to increase `-search.maxConcurrentRequests`; to increase server capacity; to reduce the number of concurrent requests

The limiter exists to stop a burst of expensive queries from exhausting memory and CPU and taking the whole node down. Rather than degrading every query, VictoriaMetrics sheds load by rejecting the overflow.

Symptoms

• Grafana panels show 503 errors or “cannot handle more than N concurrent search requests” during dashboard refreshes.
• Alerting via vmalert intermittently fails to evaluate rules, logging query 503s.
• The metric vm_concurrent_select_current sits at vm_concurrent_select_capacity.
• Query latency spikes right before the errors appear.
• Errors cluster when many users open heavy dashboards at the same moment.

Common Root Causes

• Too many simultaneous dashboards — large Grafana walls or many users refresh at once, exceeding -search.maxConcurrentRequests.
• Heavy, slow queries holding slots — high-cardinality or long-range MetricsQL queries occupy a concurrency slot for seconds each.
• Short queue duration — -search.maxQueueDuration (default 10s) is too low to ride out brief bursts.
• Undersized vmselect / node — not enough CPU and memory for the query volume.
• Alerting fan-out — vmalert evaluating many rules on a tight interval competes with interactive queries.
• Retry storms — clients that retry on 503 amplify the very overload that caused it.

Diagnostic Workflow

Check whether the read limiter is saturated. Single node uses port 8428; vmselect uses 8481:

curl -s http://localhost:8428/metrics | grep -E 'vm_concurrent_select_(current|capacity)'
curl -s http://localhost:8428/metrics | grep vm_concurrent_select_limit_timeout_total

A rising vm_concurrent_select_limit_timeout_total confirms queries are being dropped after waiting. Find the slow, slot-hogging queries with the built-in slow-query log and stats endpoint:

# Log queries slower than 5s
./victoria-metrics-prod -search.logSlowQueryDuration=5s

In vmui, open Query stats or run the top-queries API to spot the worst offenders:

curl -s 'http://localhost:8428/api/v1/status/top_queries' | head

Look at how long queries run using MetricsQL in vmui:

histogram_quantile(0.99, sum(rate(vm_request_duration_seconds_bucket{path="/api/v1/query_range"}[5m])) by (le))

Inspect the active flags:

ps aux | grep -oE '\-search.(maxConcurrentRequests|maxQueueDuration)[^ ]*'

Example Root Cause Analysis

A cluster with two vmselect pods (8 vCPU each, -search.maxConcurrentRequests=8) started returning 503s every hour on the hour. Correlating with vmalert, the team found a rule group of 120 alerts evaluating on a 1m interval — every minute it fired a wave of query_range requests that each scanned 7 days of a high-cardinality metric, holding select slots for 3–4 seconds. During the overlap with morning Grafana traffic, vm_concurrent_select_current pinned at 8 and vm_concurrent_select_limit_timeout_total climbed.

The fix was twofold: the expensive alert queries were rewritten to use recording rules (precomputed series) so each evaluation was cheap, and vmselect was scaled to -search.maxConcurrentRequests=16 with -search.maxQueueDuration=30s. Timeouts dropped to zero.

Prevention Best Practices

• Move expensive, repeated MetricsQL into recording rules so dashboards and alerts read cheap precomputed series.
• Set -search.maxConcurrentRequests in proportion to CPU cores, and raise -search.maxQueueDuration to absorb bursts.
• Cap per-query cost with -search.maxQueryDuration, -search.maxPointsPerTimeseries, and -search.maxSamplesPerQuery.
• Scale vmselect horizontally behind a load balancer for read-heavy workloads.
• Stagger Grafana refresh intervals and avoid 5s/10s auto-refresh on heavy panels.
• Alert on vm_concurrent_select_current / vm_concurrent_select_capacity > 0.8.

Quick Command Reference

# Read concurrency saturation
curl -s http://localhost:8428/metrics | grep -E 'vm_concurrent_select_(current|capacity)'

# Count of queries dropped after queueing
curl -s http://localhost:8428/metrics | grep vm_concurrent_select_limit_timeout_total

# Identify heaviest queries
curl -s 'http://localhost:8428/api/v1/status/top_queries'

# Raise limits and log slow queries
./victoria-metrics-prod -search.maxConcurrentRequests=16 \
  -search.maxQueueDuration=30s -search.logSlowQueryDuration=5s

Conclusion

This error is read-path backpressure: too many queries in flight, each holding a slot. Confirm it with vm_concurrent_select_current at capacity and a rising vm_concurrent_select_limit_timeout_total, then attack it from both ends — raise -search.maxConcurrentRequests / -search.maxQueueDuration for immediate relief, and cut the cost of your heaviest MetricsQL with recording rules and query limits so the concurrency ceiling stops being reached in the first place.