🚨 VictoriaMetrics Error Guide: 'connection refused' on vmagent remoteWrite — Restore the vminsert Path

Overview

vmagent logs this when it cannot deliver a compressed block of samples to its -remoteWrite.url target. The TCP connection is actively refused, meaning nothing is listening on that host and port:

2026-07-09T09:02:41.556Z	error	VictoriaMetrics/lib/persistentqueue/persistentqueue.go:... cannot send a block with size 1048576 bytes to "1:secret-url": Post "http://vminsert.monitoring.svc:8480/insert/0/prometheus/api/v1/write": dial tcp 10.42.3.17:8480: connect: connection refused; re-sending the block in 2.000 seconds

connection refused is specific: the target host is reachable at the network layer but no process is accepting connections on that port (versus a timeout, which points at a firewall drop, or NXDOMAIN, which points at DNS). The good news is that vmagent does not drop the data — it writes blocks to its persistent on-disk queue and keeps retrying, so as long as you fix the target before the buffer fills, no samples are lost.

Symptoms

• journalctl -u vmagent repeats connect: connection refused with re-sending the block in N seconds.
• Grafana shows a flat gap for all metrics arriving through that vmagent, starting at a sharp edge.
• The vmagent on-disk queue directory (-remoteWrite.tmpDataPath) grows steadily.
• vmagent_remotewrite_pending_data_bytes climbs while vmagent_remotewrite_requests_total{status_code="200"} stops incrementing.
• Restarting vmagent does not help — it reconnects and is refused again.
• A curl to the vminsert /insert/0/prometheus/api/v1/write endpoint from the vmagent host also returns connection refused.

Common Root Causes

• vminsert is down or crash-looping — no process is listening on 8480, so every connection is refused.
• Wrong port in -remoteWrite.url — pointing at 8428 (single-node) or 8481/8482 instead of vminsert’s 8480, or at a stale port.
• DNS resolves but the service moved — a Kubernetes Service with no ready endpoints, or an old IP cached after a pod reschedule.
• Firewall / security group / NetworkPolicy blocking 8480 between the vmagent host and vminsert (though a pure block usually times out; a RST gives “refused”).
• vminsert bound to the wrong interface — listening on 127.0.0.1:8480 instead of 0.0.0.0:8480, refusing remote clients.
• Load balancer with no healthy backends — the LB accepts nothing and resets, surfacing as refused.

Diagnostic Workflow

Confirm the error and read the exact URL vmagent is trying:

sudo journalctl -u vmagent --since '15 min ago' | grep -iE 'connection refused|remoteWrite'
ps -o cmd -C vmagent | tr ' ' '\n' | grep -i remoteWrite.url

Check whether vminsert is actually up and listening on 8480 (run on the vminsert host):

sudo systemctl status vminsert
sudo ss -ltnp | grep 8480
curl -s http://localhost:8480/metrics | grep -E 'vm_rpc|process_start_time' | head

From the vmagent host, test connectivity and the ingest endpoint directly. A refusal here confirms the target, not vmagent:

# Raw TCP reachability
nc -vz vminsert.monitoring.svc 8480
# The actual multitenant write path (accountID 0)
curl -sv -XPOST 'http://vminsert.monitoring.svc:8480/insert/0/prometheus/api/v1/write' --data-binary @/dev/null

Verify DNS resolves to a live endpoint:

getent hosts vminsert.monitoring.svc
# Kubernetes: confirm the Service has ready endpoints
kubectl -n monitoring get endpoints vminsert -o wide

Inspect vmagent’s buffer health so you know how much runway remains before data loss:

curl -s http://<vmagent-node>:8429/metrics | grep -E \
  'vmagent_remotewrite_pending_data_bytes|vmagent_remotewrite_conn_errors_total|vmagent_remotewrite_requests_total'
du -sh /var/lib/vmagent-remotewrite-data   # -remoteWrite.tmpDataPath

Check the configured disk buffer cap; if pending_data_bytes approaches -remoteWrite.maxDiskUsagePerURL, vmagent will start dropping the oldest blocks:

ps -o cmd -C vmagent | tr ' ' '\n' | grep -i maxDiskUsagePerURL

Example Root Cause Analysis

A platform team ran a VictoriaMetrics cluster in Kubernetes. After a node drain, all dashboards fed by one region went flat. vmagent logs showed dial tcp 10.42.3.17:8480: connect: connection refused repeating every two seconds, with the URL http://vminsert.monitoring.svc:8480/insert/0/prometheus/api/v1/write.

kubectl -n monitoring get endpoints vminsert showed zero ready endpoints — the vminsert pods had been evicted during the drain and were stuck Pending because a node selector no longer matched any schedulable node. The Service name still resolved (ClusterIP was intact), so DNS was fine and the failure surfaced as connection refused rather than NXDOMAIN. Meanwhile vmagent_remotewrite_pending_data_bytes had grown to 1.8 GB against a -remoteWrite.maxDiskUsagePerURL=2GB cap — minutes from dropping data.

The fix was to relax the vminsert node selector so the pods could schedule; endpoints became ready within a minute. vmagent’s next retry succeeded (vmagent_remotewrite_requests_total{status_code="200"} resumed climbing), and it drained the 1.8 GB on-disk queue over the following few minutes with zero sample loss. The team then raised -remoteWrite.maxDiskUsagePerURL to 8GB to buy more buffer for future outages and added an alert on pending bytes exceeding 50% of the cap.

Prevention Best Practices

• Run vminsert with multiple replicas behind a stable Service/LB so a single pod or node loss does not take down the ingest endpoint on 8480.
• Set -remoteWrite.maxDiskUsagePerURL generously (and provision the disk for it) so vmagent can buffer through realistic outages without dropping the oldest blocks.
• Alert on vmagent_remotewrite_pending_data_bytes crossing a fraction of the disk cap and on vmagent_remotewrite_conn_errors_total increasing — both fire before data loss.
• Pin the correct port in -remoteWrite.url: vminsert 8480, never single-node 8428 or vmselect 8481/vmstorage 8482.
• Ensure vminsert listens on 0.0.0.0:8480 (or the pod IP), not loopback, and keep firewall/NetworkPolicy rules for 8480 in version control.
• Health-check the actual write path (/insert/0/prometheus/api/v1/write) in readiness probes so unready vminsert pods are pulled from the Service before vmagent hits them.

Quick Command Reference

# See the failing remoteWrite target
sudo journalctl -u vmagent --since '15m' | grep -i 'connection refused'
ps -o cmd -C vmagent | tr ' ' '\n' | grep -i remoteWrite.url

# Is vminsert up and listening on 8480?
sudo systemctl status vminsert
sudo ss -ltnp | grep 8480

# Test the multitenant ingest path from the vmagent host (accountID 0)
nc -vz vminsert-host 8480
curl -sv -XPOST 'http://vminsert-host:8480/insert/0/prometheus/api/v1/write' --data-binary @/dev/null

# Kubernetes: are there ready endpoints?
kubectl -n monitoring get endpoints vminsert -o wide

# vmagent buffer health (vmagent self-metrics on 8429)
curl -s http://<vmagent-node>:8429/metrics | grep vmagent_remotewrite_pending_data_bytes
ps -o cmd -C vmagent | tr ' ' '\n' | grep -i maxDiskUsagePerURL

Conclusion

A vmagent connect: connection refused to ...:8480/insert/0/prometheus/api/v1/write means the vminsert endpoint has no process accepting connections — vminsert is down, listening on the wrong interface, or the Service has no ready endpoints. Start at the target: confirm vminsert is up and bound to 8480, verify DNS resolves to a live endpoint, and rule out firewall RSTs. While you fix it, vmagent is buffering to its on-disk queue, so watch vmagent_remotewrite_pending_data_bytes against -remoteWrite.maxDiskUsagePerURL — restore the path before that buffer fills and you lose zero samples. Multiple vminsert replicas and a generous disk buffer turn this class of incident into a non-event.