🚨 VictoriaMetrics Error: 'cannot execute rule ... unexpected response from datasource' — Cause, Fix, and Troubleshooting Guide

What this error means

vmalert evaluates recording and alerting rules by issuing queries against a datasource — usually vmselect (8481) or single-node victoria-metrics (8428). When a rule’s expression cannot be evaluated against that datasource, vmalert logs the group and rule name alongside the datasource’s response:

cannot execute rule "HighErrorRate" in group "app": unexpected response from datasource "http://vmselect:8481/select/0/prometheus": status code 422; response: cannot parse query

A 422 with cannot parse query means the datasource received the request but rejected the MetricsQL expression. Other status codes point elsewhere: 401/403 is auth, 404 is usually a wrong path or missing tenant prefix, and timeouts point at load. The rule and group names in the message tell you exactly which expression to fix.

What you observe in queries

• vmalert logs cannot execute rule "<name>" in group "<group>" on every evaluation interval.
• Alerts stay in an unknown or stale state; recording rules stop producing series.
• The /api/v1/rules page in vmalert shows the rule with a lastError set.
• The vmalert_execution_errors_total counter climbs for the affected group.
• Manually running the same expression in vmui returns a parse error or an auth challenge.

Store and schema causes

• Wrong -datasource.url — pointing at the wrong host, port, or path so requests never reach a working query endpoint.
• Invalid MetricsQL in a rule expr — a syntax error or unsupported function producing the 422; cannot parse query.
• Auth required to reach VictoriaMetrics — the datasource sits behind basic auth or a bearer token that vmalert is not sending.
• The datasource overloaded or timing out — vmselect is saturated and returns errors or deadline exceeded instead of results.
• Missing tenant prefix in the URL — in cluster mode the -datasource.url omits the /select/<accountID>/prometheus path.

Interrogating the store

Reproduce the exact query the rule runs, straight against the datasource URL vmalert uses:

curl -s -G 'http://vmselect:8481/select/0/prometheus/api/v1/query' \
  --data-urlencode 'query=sum(rate(http_requests_total{status=~"5.."}[5m]))'

A 422 here confirms the expression, not vmalert, is at fault. Inspect the configured datasource and tenant path:

ps aux | grep vmalert | grep -oE '\-datasource.url[^ ]*'

Validate every rule file offline before restarting the process:

./vmalert -rule='/etc/vmalert/rules/*.yml' \
  -datasource.url='http://vmselect:8481/select/0/prometheus' \
  -dryRun

Resolution

1. Fix -datasource.url (include the tenant prefix for cluster mode). For a clustered read path the URL must carry the account/tenant path:

./vmalert -datasource.url='http://vmselect:8481/select/0/prometheus' \
  -rule='/etc/vmalert/rules/*.yml'

Single-node deployments use http://victoria-metrics:8428 with no /select/<id> prefix.

2. Validate rules with vmalert -dryRun. Run -dryRun in CI so a malformed expr is caught before it reaches production and starts logging cannot parse query.

3. Supply credentials with -datasource.basicAuth.username/-datasource.basicAuth.password or a bearer token. If VictoriaMetrics is behind auth, give vmalert matching credentials:

./vmalert -datasource.url='https://metrics.example.com/select/0/prometheus' \
  -datasource.basicAuth.username='vmalert' \
  -datasource.basicAuth.password='***' \
  -rule='/etc/vmalert/rules/*.yml'

4. Add -remoteWrite.url so alert and recording state persists across restarts and is queryable, pointing it at vminsert (8480) or single-node (8428).

5. Check network and datasource load. If the error is intermittent with timeouts rather than 422, the datasource is saturated — see the context deadline and max concurrent requests guides.

Sustainable retention settings

• The status code is the fastest triage signal: 422 is a bad query, 401/403 is auth, 404 is usually a missing tenant prefix — read it before touching the rule.
• In cluster mode the read path (/select/<id>/prometheus) and write path (/insert/<id>/prometheus) differ; -datasource.url and -remoteWrite.url must each use the correct one.
• -dryRun validates syntax but still needs a reachable datasource to check that referenced metrics resolve.
• Rotating datasource credentials without updating vmalert flips every rule in the group to an error at once.

Related store errors

• VictoriaMetrics Error Guide: cannot parse MetricsQL
• VictoriaMetrics Error Guide: context deadline exceeded
• VictoriaMetrics Error Guide: max concurrent requests