🚨 VictoriaMetrics Error: 'cannot unmarshal labels ... the data is corrupted' — Cause, Fix, and Troubleshooting Guide

Overview

vminsert (in a cluster) and single-node victoria-metrics parse each incoming ingestion block by reading a length-prefixed, compact binary layout — first the size of a string, then that many bytes. When the bytes on the wire do not match the layout the decoder expects, it cannot read a label and aborts with:

cannot unmarshal labels: cannot unmarshal string size: cannot read enough data; probably the data is corrupted

Despite the “corrupted” wording, the data is rarely corrupt on disk. It means the received bytes are not the format the endpoint expects — usually because a mismatched or wrong client is talking to the native ingestion port, the request landed on the wrong path, or something in the middle rewrote the body. Treat it as a protocol/path problem first, not a storage fault.

Symptoms

• vminsert or single-node logs cannot unmarshal labels ... probably the data is corrupted on every batch from one source.
• Ingestion from a specific vmagent, exporter, or client fails while other sources ingest normally.
• The failures started right after upgrading vmagent or the cluster components (but not both together).
• A custom or third-party client began posting to the native ingestion port.
• Metrics from an app go missing and its remote_write client logs repeated non-2xx responses.

Common Root Causes

• Version skew between vmagent and the cluster — a vmagent on a much newer or older release than vminsert/vmstorage can emit a native block the receiver cannot decode.
• Wrong client on the native port — a non-VictoriaMetrics client (or a generic HTTP tool) POSTing arbitrary bytes to the native ingestion endpoint, which VM tries to parse as a native block.
• Remote_write sent to the wrong path/port — Prometheus remote_write aimed at the native ingestion path instead of /insert/<tenant>/prometheus/api/v1/write.
• A proxy mangling the body — an ingress, load balancer, or WAF that rewrites, re-compresses, buffers, or truncates the request body so the length prefixes no longer match the payload.

How to diagnose

Compare versions across every component in the ingest path — skew is the most common cause:

# vmagent version
./vmagent -version
# cluster receiver version
curl -s http://localhost:8480/metrics | grep -m1 vm_app_version

Identify which source is failing and where it is sending. On the vmagent side, inspect the configured URL:

ps aux | grep vmagent | grep -oE '\-remoteWrite.url[^ ]*'

Prometheus-style remote_write must target the write path, not the native port. A correct cluster URL looks like:

http://vminsert:8480/insert/0/prometheus/api/v1/write

Send one known-good sample directly to the cluster, bypassing any proxy, to prove the endpoint itself is fine:

# Line-protocol import to single-node (8428) — should return HTTP 204
curl -s -w '%{http_code}\n' \
  'http://localhost:8428/api/v1/import/prometheus' \
  -d 'test_metric{job="diag"} 1'

If this succeeds directly but fails through your load balancer, the proxy is altering the body.

Fixes

1. Align component versions. Keep vmagent and the cluster (vminsert/vmselect/vmstorage) on matching releases, and upgrade them together. Pin the same tag across your deployment so a rolling upgrade never leaves a newer vmagent talking to older vminsert.

2. Send remote_write to the correct path and port. Point Prometheus/vmagent remote_write at the write endpoint on vminsert 8480, not the native ingestion port:

-remoteWrite.url=http://vminsert:8480/insert/0/prometheus/api/v1/write

The 0 is the tenant (accountID); use your real tenant. Single-node uses http://victoria-metrics:8428/api/v1/write.

3. Verify -remoteWrite.url on every client. Audit each vmagent and Prometheus remote_write block for a stray path or port. A client posting Prometheus remote_write bytes to the native endpoint will always trip this error.

4. Bypass or fix a body-rewriting proxy. Route ingestion straight to vminsert to confirm the proxy is the culprit, then reconfigure it to pass the body through untouched — disable response/request buffering transforms, re-compression, and any WAF rule that edits the payload. Ensure Content-Encoding is preserved end to end.

5. Point third-party clients at a text endpoint. If a non-VM client must ingest, send it to a documented text/line-protocol path (for example /api/v1/import/prometheus) rather than the native binary port, so VM parses it with the matching decoder.

What to watch out for

• “Probably the data is corrupted” is misleading — check protocol, path, and versions before you suspect disk or memory corruption.
• Roll upgrades so vmagent is never far ahead of or behind vminsert/vmstorage; large version gaps are the usual trigger.
• Anything in the request path that touches the body (buffering, compression, rewriting) can break the length-prefixed format — keep ingestion proxies as transparent pass-throughs.
• One failing source can hide among many healthy ones; use the client-side non-2xx logs to pinpoint which vmagent or exporter is at fault.

Related

• VictoriaMetrics Error Guide: connection reset by peer
• VictoriaMetrics Error Guide: missing tenant id