🚨 VictoriaMetrics Error: 'unexpected end of stream' — Cause, Fix, and Troubleshooting Guide

Overview

VictoriaMetrics streams remote-write blocks from vmagent to vminsert over a persistent connection. When that connection dies mid-block — before vminsert has read the whole payload — the protoparser gives up and logs:

cannot read block from vmagent: unexpected end of stream; the connection was probably closed while transferring data

This is a transport-level error, not a data-validity error. vminsert had a partial block and no clean end-of-stream marker, so it discarded the read. The common culprit is something between vmagent and vminsert — a load balancer, proxy, or the peer process itself — closing the socket while bytes were still in flight. vmagent normally retries, so a low rate of these is cosmetic; a steady stream means a real path or capacity problem.

Symptoms

• vminsert logs repeated cannot read block from vmagent: unexpected end of stream lines.
• vmagent logs remote-write retries and rising in-memory or on-disk queue size.
• Ingest throughput dips or stalls in bursts that line up with the error timestamps.
• The errors cluster right after a vminsert or vmagent restart, or an OOM kill on either side.
• A load balancer or service mesh sits on the write path and has a short idle/read timeout.

Common Root Causes

• LB or proxy idle timeout — an L4/L7 load balancer between vmagent and vminsert closes connections it considers idle, cutting off a slow or bursty transfer.
• Peer restart or OOM kill — vminsert (or vmagent) was restarted, rescheduled, or OOM-killed mid-send, so the socket dropped with a block half-transferred.
• Truncated payload — the connection was reset by a network blip and the block arrived incomplete.
• Node/network instability — MTU mismatch, missing TCP keepalive, or a flaky link tears down long-lived connections.

How to diagnose

Confirm whether the errors line up with restarts or OOM events on the receiving side:

# On the vminsert host/node — look for OOM kills and process restarts
dmesg -T | grep -iE 'oom|killed process'
# In Kubernetes, check restart counts and last-terminated reason
kubectl get pods -l app=vminsert -o wide
kubectl describe pod <vminsert-pod> | grep -A3 'Last State'

Check the vmagent side for a growing retry/queue backlog that proves the sends are not completing:

# vmagent persistent-queue and retry metrics
curl -s 'http://localhost:8429/metrics' \
  | grep -E 'vmagent_remotewrite_(retries_count|pending_bytes|packets_dropped)_total'

If an LB or proxy is on the path, inspect its idle/read timeout and its own connection-reset counters. Correlate the LB’s connection-close logs with the vminsert error timestamps to confirm the middlebox is the one hanging up.

Fixes

1. Raise idle and read timeouts on every hop. Any LB, reverse proxy, or service mesh between vmagent and vminsert must allow long-lived streaming connections. Set the idle/read timeout well above your largest block flush interval so a bursty transfer is never cut short.

2. Fix the peer that is dropping. If dmesg or kubectl describe shows OOM kills or restarts on vminsert, give it more memory (or lower its ingest concurrency) so it stops dying mid-block. Stabilize the process first; the stream errors follow it.

3. Lean on vmagent retries and queueing. vmagent buffers and retries failed sends, so a transient reset is recoverable. Make sure the persistent queue has room and tune the send concurrency:

./vmagent -remoteWrite.url=http://vminsert:8480/insert/0/prometheus/api/v1/write \
  -remoteWrite.queues=8 \
  -remoteWrite.tmpDataPath=/var/lib/vmagent

4. Harden the network path. Verify a consistent MTU end to end (avoid black-hole fragmentation) and confirm TCP keepalive is enabled so idle-but-alive connections are not silently reaped by an intermediate device.

What to watch out for

• A low, steady rate of these after routine restarts is usually harmless because vmagent retries — alert on rate and sustained queue growth, not on a single line.
• Raising timeouts on vminsert alone does nothing if a load balancer in the middle still has a short one; fix every hop.
• OOM-driven restarts masquerade as network errors — always rule out peer crashes before blaming the network.
• If vmagent’s persistent queue fills, it starts dropping data; watch vmagent_remotewrite_packets_dropped_total alongside this error.

Related

• VictoriaMetrics Error Guide: connection reset by peer
• VictoriaMetrics Error Guide: context deadline exceeded
• VictoriaMetrics Error Guide: remote write connection refused