🚨 VictoriaMetrics Error: 'unexpected status code received after sending a block ... 400' — Cause, Fix, and Troubleshooting Guide

Overview

When vmagent ships a batch of samples to remote storage, it expects a 2xx acknowledgement. A 400 Bad Request means the receiver parsed the request far enough to reject it as malformed and refused the whole block. vmagent logs the failure with the destination URL and the receiver’s response body:

unexpected status code received after sending a block to "http://vminsert:8480/insert/0/prometheus/api/v1/write": 400; response body: "cannot unmarshal MetricRow"

The important part is the quoted response body — that text comes straight from the receiver (vminsert or single-node victoria-metrics) and names the actual reason. A 400 is a client-side problem: retrying unchanged will not help, because the request itself is wrong. vmagent will keep buffering and re-sending, so this error tends to repeat until you fix the cause.

Symptoms

• vmagent logs repeat the unexpected status code ... : 400 line for the same remote-write URL.
• The remote-write queue backs up and vmagent_remotewrite_pending_data_bytes climbs on the affected URL.
• Data written through a different, correctly configured path arrives fine, but this destination stays empty.
• The receiver’s own logs show a matching rejection at the same timestamps.
• The failure started after a config change, a version bump, or a tenant/path edit — not spontaneously.

Common Root Causes

• The receiver rejected a malformed request — a bad tenant path, invalid or over-limit labels, or a corrupt/mis-encoded body produces cannot unmarshal MetricRow.
• Sending to a non-write endpoint — the URL points at a query or UI path instead of the .../api/v1/write ingestion path, so the receiver can’t parse the block.
• A version mismatch — vmagent and the receiver are far enough apart that the wire format or path expectations differ, and the receiver rejects the payload.

How to diagnose

Start with the response body vmagent already logged — it is the receiver’s verdict. Then reproduce the write directly so you can see the receiver’s status and body without vmagent in the way:

# Send a single well-formed sample to the same URL and read the status + body
curl -s -w '\nHTTP %{http_code}\n' \
  -H 'Content-Type: text/plain' \
  --data-binary 'test_metric{job="probe"} 1' \
  'http://vminsert:8480/insert/0/prometheus/api/v1/import/prometheus'

Check the receiver’s own logs for the rejection that pairs with vmagent’s 400 — it usually spells out the offending label or path:

# vminsert (cluster) or victoria-metrics (single-node) logs
journalctl -u vminsert --since '10 min ago' | grep -Ei 'reject|unmarshal|invalid'

Confirm both sides are on compatible builds:

curl -s http://vmagent:8429/metrics | grep vm_app_version
curl -s http://vminsert:8480/metrics | grep vm_app_version

Fixes

1. Read the response body first. The quoted text after response body: is authoritative — cannot unmarshal MetricRow points at a corrupt or mis-encoded block, while messages about labels or tenants point at the URL or payload. Fix what it names before changing anything else.

2. Fix the -remoteWrite.url path and tenant prefix. Ensure the URL is the ingestion path with the correct tenant, not a query or UI path:

# Correct cluster write path (tenant 0), not a /select or /vmui path
./vmagent -remoteWrite.url=http://vminsert:8480/insert/0/prometheus/api/v1/write

3. Align vmagent and receiver versions. Upgrade the lagging component so both speak the same remote-write format:

# Pin both to the same VictoriaMetrics release train
docker pull victoriametrics/vmagent:v1.xxx.x
docker pull victoriametrics/vminsert:v1.xxx.x

4. Inspect the receiver logs for the matching rejection. Whichever component returns the 400 — vminsert in a cluster or single-node victoria-metrics — logs the concrete reason (bad label, oversized row, wrong tenant). Fix that at the source, then vmagent’s buffered block drains on the next flush.

What to watch out for

• A 400 is never fixed by retrying — vmagent retries automatically, so if the log keeps repeating, the request is still wrong.
• Don’t confuse a 400 (malformed, client’s fault) with a 429 (rate limited) or 5xx (receiver overloaded) — they need opposite responses.
• The insert/<tenant>/... prefix is easy to get wrong; a valid-looking but nonexistent tenant path can surface here rather than at startup.
• Long-lived 400s can fill the persistent queue on disk; clearing the cause lets vmagent flush the backlog, which may briefly spike ingestion.

Related

• VictoriaMetrics Error Guide: remote write 429
• VictoriaMetrics Error Guide: cannot unmarshal labels
• VictoriaMetrics Error Guide: unsupported path requested