🚨 VictoriaMetrics Error Guide: 'cannot execute query on all the vmstorage nodes; the following nodes are unavailable' — Restore vmselect-to-vmstorage Connectivity

Overview

In a VictoriaMetrics cluster, vmselect fans every query out to all vmstorage nodes over the query port (8401). When it cannot reach one or more of them, it aborts the query and reports exactly which nodes were unreachable:

cannot execute query on all the vmstorage nodes; the following nodes are unavailable: 10.0.3.11:8401, 10.0.3.12:8401

A partial-failure variant appears when only some nodes are down but vmselect is configured to require all of them (-search.denyPartialResponse):

cannot perform search on all the vmstorage nodes because some of them are unavailable and -search.denyPartialResponse is set to true

The message is a connectivity and health report, not a query-logic error. vmselect is telling you it opened the list of -storageNode addresses, tried each on port 8401, and one or more did not answer. The fix is to restore the unhealthy vmstorage nodes: confirm they are running, listening on 8401, and reachable across the network from vmselect.

Symptoms

• Queries fail in Grafana and vmui with a list of host:8401 addresses marked unavailable.
• Only queries fail; ingestion may still succeed if vminsert (port 8480) can reach the surviving nodes.
• With -search.denyPartialResponse=true, a single down node fails the whole query; with it false, results silently miss the down node’s data.
• vmselect logs repeatedly show cannot dial or connection refused to :8401 addresses.
• The listed nodes match vmstorage pods/hosts that recently restarted, were rescheduled, or lost network.
• vm_rpc_* error metrics on vmselect climb, and target-down alerts fire for the affected vmstorage.

Common Root Causes

• vmstorage process down — crashed, OOM-killed, or stopped for maintenance, so nothing listens on 8401.
• Network partition / firewall — a security group, NetworkPolicy, or firewall rule blocks vmselect -> vmstorage:8401.
• Wrong -storageNode addresses — vmselect (and vminsert) configured with stale IPs/DNS that no longer resolve to live nodes.
• DNS / service-discovery lag — in Kubernetes a rescheduled vmstorage pod got a new IP that the headless service or config hasn’t caught up to.
• Port confusion — using 8482 (HTTP) instead of 8401 (vmselect RPC), or 8400 (vminsert RPC), in the -storageNode list.
• vmstorage overloaded — disk or CPU saturation makes it stop answering RPC in time, appearing as unavailable.
• Version skew — incompatible vmstorage/vmselect versions failing the RPC handshake.

Diagnostic Workflow

Start by confirming which -storageNode addresses vmselect is actually using (port 8481 is vmselect’s HTTP):

curl -s http://<vmselect>:8481/flags | grep -i 'storageNode\|denyPartialResponse'

Test raw TCP reachability from the vmselect host to every vmstorage on the RPC port 8401:

for n in 10.0.3.11 10.0.3.12 10.0.3.13; do nc -zv $n 8401; done

Verify each vmstorage is alive and healthy on its HTTP port (8482). A healthy node answers /health and exposes metrics:

for n in 10.0.3.11 10.0.3.12 10.0.3.13; do
  echo "== $n =="; curl -s -o /dev/null -w '%{http_code}\n' http://$n:8482/health
done
curl -s http://10.0.3.11:8482/metrics | grep -E 'vm_free_disk_space_bytes|process_resident_memory_bytes|vm_rows '

Confirm vmstorage is actually listening on the RPC port on the node itself:

ss -tlnp | grep -E '8401|8400|8482'

Read the logs on both sides. vmselect shows the dial failures; vmstorage shows whether it is crash-looping:

journalctl -u vmselect --since '15 min ago' | grep -i 'unavailable\|cannot dial\|vmstorage'
journalctl -u vmstorage --since '15 min ago' | grep -i 'FATAL\|panic\|started'

Check vmselect’s own metrics for RPC errors toward storage:

curl -s http://<vmselect>:8481/metrics | grep -E 'vm_rpc_connection_errors_total|vm_rpc_dial_errors_total'

In Kubernetes, verify the vmstorage pods are Ready and the headless service resolves to their current IPs:

kubectl get pods -l app=vmstorage -o wide
kubectl get endpoints vmstorage

If TCP to 8401 succeeds but the query still fails, suspect version skew or an overloaded node timing out — cross-check vmstorage CPU, disk, and vm_free_disk_space_bytes (a node in read-only mode still serves reads, so that is not the cause of unavailability, but saturation is).

Example Root Cause Analysis

A cluster started returning cannot execute query on all the vmstorage nodes; the following nodes are unavailable: 10.0.3.12:8401 for every dashboard, while ingestion continued fine.

From the vmselect host, nc -zv 10.0.3.12 8401 returned connection refused, while .11 and .13 connected. curl http://10.0.3.12:8482/health also failed, confirming the whole vmstorage process was down, not just the RPC port. journalctl -u vmstorage on that node showed a FATAL at startup — cannot obtain lock on file — because a hard reboot had left a stale process; the node was crash-looping and never bound 8401.

Because vmselect ran with -search.denyPartialResponse=true, a single missing node failed every query rather than returning partial data — which is why all dashboards broke, not just those needing shard .12. After clearing the stale lock and letting vmstorage start cleanly, it bound 8401/8482, nc connected, /health returned 200, and vmselect’s vm_rpc_dial_errors_total stopped climbing. Queries recovered immediately.

The team also documented that denyPartialResponse=true is correct for their correctness-sensitive use case, but it makes a single node’s health a cluster-wide query dependency — so vmstorage availability alerting was tightened.

Prevention Best Practices

• Alert on every vmstorage /health and on vm_rpc_dial_errors_total / vm_rpc_connection_errors_total growing on vmselect, so a down node pages before dashboards break.
• In Kubernetes, run vmstorage as a StatefulSet with a headless service and address it by stable DNS names, so rescheduled pods don’t leave vmselect pointing at dead IPs.
• Keep -storageNode lists in sync across vmselect and vminsert; automate them from service discovery rather than hardcoding IPs.
• Open and monitor 8401 (vmselect RPC), 8400 (vminsert RPC), and 8482 (HTTP) between the right components; document them so firewall/NetworkPolicy changes don’t silently break connectivity.
• Decide deliberately on -search.denyPartialResponse: true for correctness (but any node down fails queries), false for availability (queries return partial data with a warning).
• Run matching vmstorage/vmselect/vminsert versions to avoid RPC handshake failures.
• Give vmstorage enough CPU/disk headroom so saturation never makes it look “unavailable” to vmselect.

Quick Command Reference

# What storage nodes does vmselect target, and is partial response denied?
curl -s http://<vmselect>:8481/flags | grep -i 'storageNode\|denyPartialResponse'

# TCP reachability to each vmstorage RPC port
for n in <ip1> <ip2> <ip3>; do nc -zv $n 8401; done

# vmstorage health + key metrics (HTTP port 8482)
curl -s -o /dev/null -w '%{http_code}\n' http://<vmstorage>:8482/health
curl -s http://<vmstorage>:8482/metrics | grep -E 'vm_free_disk_space_bytes|process_resident_memory_bytes'

# Is vmstorage listening locally?
ss -tlnp | grep -E '8401|8400|8482'

# RPC error metrics on vmselect
curl -s http://<vmselect>:8481/metrics | grep -E 'vm_rpc_dial_errors_total|vm_rpc_connection_errors_total'

# Logs
journalctl -u vmstorage --since '15 min ago' | grep -i 'FATAL\|started'

# Kubernetes: pods and endpoints
kubectl get pods -l app=vmstorage -o wide
kubectl get endpoints vmstorage

Conclusion

cannot execute query on all the vmstorage nodes; the following nodes are unavailable is vmselect telling you precisely which vmstorage nodes it could not reach on port 8401. Treat it as a health-and-connectivity checklist: confirm each named node’s process is running, that it is listening on 8401/8482, and that the network path from vmselect is open. Watch for stale -storageNode addresses after pod reschedules, and understand that -search.denyPartialResponse=true turns any single node’s outage into a cluster-wide query failure. Fix the unhealthy node, keep storage-node lists in sync with service discovery, and alert on RPC errors so the next outage is caught before your dashboards go dark.