🚨 VictoriaMetrics Error: 'cannot execute rule ... unexpected response from datasource' — Cause, Fix, and Troubleshooting Guide

Overview

vmalert evaluates recording and alerting rules by issuing queries against a datasource — usually vmselect (8481) or single-node victoria-metrics (8428). When a rule’s expression cannot be evaluated against that datasource, vmalert logs the group and rule name alongside the datasource’s response:

cannot execute rule "HighErrorRate" in group "app": unexpected response from datasource "http://vmselect:8481/select/0/prometheus": status code 422; response: cannot parse query

A 422 with cannot parse query means the datasource received the request but rejected the MetricsQL expression. Other status codes point elsewhere: 401/403 is auth, 404 is usually a wrong path or missing tenant prefix, and timeouts point at load. The rule and group names in the message tell you exactly which expression to fix.

Symptoms

• vmalert logs cannot execute rule "<name>" in group "<group>" on every evaluation interval.
• Alerts stay in an unknown or stale state; recording rules stop producing series.
• The /api/v1/rules page in vmalert shows the rule with a lastError set.
• The vmalert_execution_errors_total counter climbs for the affected group.
• Manually running the same expression in vmui returns a parse error or an auth challenge.

Common Root Causes

• Wrong -datasource.url — pointing at the wrong host, port, or path so requests never reach a working query endpoint.
• Invalid MetricsQL in a rule expr — a syntax error or unsupported function producing the 422; cannot parse query.
• Auth required to reach VictoriaMetrics — the datasource sits behind basic auth or a bearer token that vmalert is not sending.
• The datasource overloaded or timing out — vmselect is saturated and returns errors or deadline exceeded instead of results.
• Missing tenant prefix in the URL — in cluster mode the -datasource.url omits the /select/<accountID>/prometheus path.

How to diagnose

Reproduce the exact query the rule runs, straight against the datasource URL vmalert uses:

curl -s -G 'http://vmselect:8481/select/0/prometheus/api/v1/query' \
  --data-urlencode 'query=sum(rate(http_requests_total{status=~"5.."}[5m]))'

A 422 here confirms the expression, not vmalert, is at fault. Inspect the configured datasource and tenant path:

ps aux | grep vmalert | grep -oE '\-datasource.url[^ ]*'

Validate every rule file offline before restarting the process:

./vmalert -rule='/etc/vmalert/rules/*.yml' \
  -datasource.url='http://vmselect:8481/select/0/prometheus' \
  -dryRun

Fixes

1. Fix -datasource.url (include the tenant prefix for cluster mode). For a clustered read path the URL must carry the account/tenant path:

./vmalert -datasource.url='http://vmselect:8481/select/0/prometheus' \
  -rule='/etc/vmalert/rules/*.yml'

Single-node deployments use http://victoria-metrics:8428 with no /select/<id> prefix.

2. Validate rules with vmalert -dryRun. Run -dryRun in CI so a malformed expr is caught before it reaches production and starts logging cannot parse query.

3. Supply credentials with -datasource.basicAuth.username/-datasource.basicAuth.password or a bearer token. If VictoriaMetrics is behind auth, give vmalert matching credentials:

./vmalert -datasource.url='https://metrics.example.com/select/0/prometheus' \
  -datasource.basicAuth.username='vmalert' \
  -datasource.basicAuth.password='***' \
  -rule='/etc/vmalert/rules/*.yml'

4. Add -remoteWrite.url so alert and recording state persists across restarts and is queryable, pointing it at vminsert (8480) or single-node (8428).

5. Check network and datasource load. If the error is intermittent with timeouts rather than 422, the datasource is saturated — see the context deadline and max concurrent requests guides.

What to watch out for

• The status code is the fastest triage signal: 422 is a bad query, 401/403 is auth, 404 is usually a missing tenant prefix — read it before touching the rule.
• In cluster mode the read path (/select/<id>/prometheus) and write path (/insert/<id>/prometheus) differ; -datasource.url and -remoteWrite.url must each use the correct one.
• -dryRun validates syntax but still needs a reachable datasource to check that referenced metrics resolve.
• Rotating datasource credentials without updating vmalert flips every rule in the group to an error at once.

Related

• VictoriaMetrics Error Guide: cannot parse MetricsQL
• VictoriaMetrics Error Guide: context deadline exceeded
• VictoriaMetrics Error Guide: max concurrent requests