🚨 VictoriaMetrics Error: 'cannot decode snappy-encoded remote_write request body: snappy: corrupt input' — Cause, Fix, and Troubleshooting Guide

Overview

The Prometheus remote_write protocol sends a Protobuf payload compressed with Snappy. VictoriaMetrics’ write receiver — vminsert or single-node victoria-metrics at /api/v1/write — expects exactly that. When the body it receives is not valid Snappy, it cannot decompress it and returns:

cannot decode snappy-encoded remote_write request body: snappy: corrupt input

This is a transport/encoding mismatch, not a data or cardinality problem. The receiver got bytes it cannot interpret as a Snappy stream — because the client did not Snappy-compress the body, a proxy altered it in transit, the wrong endpoint was hit, or the body was compressed twice. The write is rejected before any samples are parsed.

Symptoms

• /api/v1/write requests fail with snappy: corrupt input in vminsert/single-node logs.
• A specific client or proxy path fails while direct writes from vmagent/Prometheus succeed.
• Writes started failing right after inserting or reconfiguring a proxy or API gateway.
• A custom or non-remote-write client posting to the write endpoint gets rejected immediately.
• No samples land in storage even though the client reports it is “sending” data.

Common Root Causes

• Client is not snappy-compressing the payload — a custom client posts raw or gzip data to a snappy endpoint.
• A proxy decompressed or re-encoded the body — an NGINX/API gateway rewrote or buffered-and-altered the request body.
• Wrong Content-Encoding header — the header claims an encoding that does not match the actual bytes.
• A non-remote-write client hit the write endpoint — something posts JSON/line-protocol to /api/v1/write.
• Double compression — the already-snappy body was gzip-wrapped again by the client or a proxy.

How to diagnose

Confirm what the client is actually sending. A correct Prometheus remote_write uses snappy and the remote-write protobuf content type:

# Inspect the client's real request headers and body encoding
curl -si -X POST 'http://localhost:8480/insert/0/prometheus/api/v1/write' \
  -H 'Content-Encoding: snappy' \
  -H 'Content-Type: application/x-protobuf' \
  --data-binary @sample-snappy.bin | head

Bypass the proxy and write straight to the receiver to isolate whether the proxy is mangling the body:

# If direct-to-vminsert works but through-proxy fails, the proxy is the cause
tcpdump -A -s0 -i any 'tcp port 8480 and (tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504f5354)' | head

Check the client/agent config for the endpoint it targets:

grep -nE 'remote_write|url|remoteWrite.url' /etc/prometheus/prometheus.yml /etc/vmagent/*.yml 2>/dev/null

Fixes

1. Send standard Prometheus remote_write (snappy) from Prometheus or vmagent, which handle the encoding for you:

# Prometheus
remote_write:
  - url: http://vminsert:8480/insert/0/prometheus/api/v1/write

# vmagent forwards snappy remote_write natively
./vmagent -remoteWrite.url=http://vminsert:8480/insert/0/prometheus/api/v1/write

2. Ensure any proxy passes the body through untouched. Disable request buffering/rewriting so the snappy body reaches VictoriaMetrics byte-for-byte:

location /insert/ {
    proxy_pass http://vminsert:8480;
    proxy_request_buffering off;
    proxy_set_header Content-Encoding $http_content_encoding;
}

3. Point clients at the correct write endpoint. Use /api/v1/write (cluster: /insert/<tenant>/prometheus/api/v1/write); do not post line protocol or JSON there.

4. Do not gzip the already-snappy body. Remove any extra compression layer added by the client or proxy so the payload is snappy, not gzip-over-snappy.

What to watch out for

• Custom exporters that “speak remote_write” often skip the Snappy step — verify with a known-good sender (vmagent) before blaming the server.
• API gateways love to buffer and re-encode bodies; a working setup can break the moment a proxy is inserted, so test direct-to-receiver first.
• Match Content-Encoding to the actual bytes — a mismatched header is as fatal as truly corrupt data.
• VictoriaMetrics offers other ingestion endpoints (InfluxDB, Graphite, JSON import); route non-remote-write clients to those instead of /api/v1/write.

Related

• VictoriaMetrics Error Guide: remote write 429
• VictoriaMetrics Error Guide: connection reset by peer