🚨 VictoriaMetrics Error Guide: vmagent 'connection reset by peer' to vmstorage — Restore the Ingestion Path

Overview

In cluster mode, vminsert talks to vmstorage over a binary protocol on the -vminsertAddr port (default :8400), and vmselect uses -vmselectAddr (default :8401). When that TCP connection is severed mid-stream, the sender logs:

cannot send data to vmstorage "10.0.3.7:8400": cannot read data from client: read tcp 10.0.2.4:52344->10.0.3.7:8400: read: connection reset by peer

vmagent shows the same class of error when its remote_write target drops the connection. “connection reset by peer” is a TCP RST: the other side closed the socket abruptly — typically because the vmstorage process died, restarted, was killed, or a network device dropped the flow.

Symptoms

• vminsert/vmagent logs stream connection reset by peer toward one or more vmstorage addresses.
• Ingestion drops or stalls; vm_rpc_connection_errors_total climbs.
• One vmstorage node flaps in and out; vminsert reroutes writes to healthy nodes.
• vmselect queries return partial data or cannot fetch data from vmstorage nodes while a node is down.
• Errors coincide with vmstorage restarts, OOM kills, or deploys.

Common Root Causes

• vmstorage OOM or crash — the process was killed (often OOMKilled in Kubernetes), resetting all in-flight connections.
• vmstorage restart/rollout — a rolling update or config reload cycles the process and drops connections.
• Wrong or blocked ports — -vminsertAddr (8400) / -vmselectAddr (8401) not reachable due to a firewall, SecurityGroup, or NetworkPolicy.
• Network instability — MTU mismatch, LB idle timeouts, or a flaky overlay network cutting long-lived TCP flows.
• Version skew — vminsert/vmselect and vmstorage on incompatible versions rejecting the RPC handshake.
• Resource exhaustion on vmstorage — too many open files or CPU starvation causing the process to drop clients.

Diagnostic Workflow

Verify each vmstorage node is alive and its metrics port (8482) responds:

for n in vmstorage-1 vmstorage-2 vmstorage-3; do
  echo "$n:"; curl -s -o /dev/null -w '%{http_code}\n' http://$n:8482/health
done

Check whether vmstorage restarted or was OOMKilled:

kubectl get pods -l app=vmstorage -o wide
kubectl describe pod vmstorage-1 | grep -iE 'restart|oomkilled|last state'
journalctl -u vmstorage --since '30 min ago' | grep -iE 'signal|killed|panic'

Confirm the binary protocol ports are open from the sender:

nc -zv vmstorage-1 8400   # vminsert -> vmstorage
nc -zv vmstorage-1 8401   # vmselect -> vmstorage

Look at RPC error counters on vminsert (port 8480) and check vmstorage memory pressure:

curl -s http://vminsert:8480/metrics | grep -E 'vm_rpc_(connection_errors_total|dial_errors_total)'
curl -s http://vmstorage-1:8482/metrics | grep -E 'process_resident_memory_bytes|vm_rows{'

Confirm all components run the same VictoriaMetrics version:

curl -s http://vminsert:8480/metrics | grep vm_app_version
curl -s http://vmstorage-1:8482/metrics | grep vm_app_version

Example Root Cause Analysis

A Kubernetes cluster running VictoriaMetrics cluster mode logged bursts of connection reset by peer from all three vminsert pods to vmstorage-2:8400 every few hours. kubectl describe pod vmstorage-2 showed Last State: Terminated, Reason: OOMKilled. The node had a 2Gi memory limit but ingestion had grown, and periodic background merges pushed process_resident_memory_bytes past the limit, so the kubelet killed it and every in-flight insert connection was reset.

The fix was raising the vmstorage memory limit to 6Gi, setting -memory.allowedPercent=60 so VictoriaMetrics sized its caches to the container limit, and adding a PodDisruptionBudget so rollouts cycled one node at a time. After that, vm_rpc_connection_errors_total flatlined and ingestion stopped stalling.

Prevention Best Practices

• Right-size vmstorage memory and set -memory.allowedPercent to the container limit so caches never push the process into an OOM kill.
• Roll out cluster components one node at a time with a PodDisruptionBudget; expect brief resets during restarts and let vminsert reroute.
• Keep vminsert, vmselect, and vmstorage on the same VictoriaMetrics version to avoid RPC handshake failures.
• Verify NetworkPolicies/SecurityGroups allow ports 8400 and 8401 between components.
• Raise LB/proxy idle timeouts above VictoriaMetrics’ keep-alive so long-lived RPC connections are not cut.
• Alert on vm_rpc_connection_errors_total and vmstorage restart count.

Quick Command Reference

# vmstorage liveness
curl -s http://vmstorage-1:8482/health

# Was it OOMKilled / restarted?
kubectl describe pod vmstorage-1 | grep -iE 'restart|oomkilled|last state'

# Binary protocol reachability
nc -zv vmstorage-1 8400   # insert path
nc -zv vmstorage-1 8401   # select path

# RPC error counters on vminsert
curl -s http://vminsert:8480/metrics | grep vm_rpc_connection_errors_total

# Size caches to the container limit
./vmstorage-prod -memory.allowedPercent=60

Conclusion

“connection reset by peer” between VictoriaMetrics components is a symptom, not a root cause: some vmstorage node closed the socket. The overwhelming majority of cases are an OOM kill or a restart, so start with kubectl describe/journalctl and the vm_rpc_connection_errors_total counter. Once you confirm whether it is memory, version skew, or blocked 8400/8401 ports, fix that underlying issue — cluster mode is designed to reroute around a briefly-missing node, so a healthy, right-sized vmstorage fleet makes these resets disappear.