🚨 VictoriaMetrics Error Guide: remote_write '429 Too Many Requests' — Relieve Ingestion Throttling

Overview

A remote_write client — vmagent or Prometheus — receives HTTP 429 Too Many Requests from VictoriaMetrics when the ingestion endpoint is throttling or shedding load. vmagent surfaces it in its logs:

cannot send block with size 1048576 bytes to "1:secret-url": unexpected status code returned when sending data to remote storage: 429 Too Many Requests; response body: "too many outstanding insert requests..."

A 429 is retriable backpressure: the write was not accepted right now, and the client should slow down and retry. It is the read side of the same coin as insert saturation — VictoriaMetrics is protecting itself by asking senders to back off rather than accepting data it cannot persist.

Symptoms

• vmagent logs repeated 429 Too Many Requests when flushing blocks to -remoteWrite.url.
• The vmagent persistent queue on disk grows (vmagent_remotewrite_pending_data_bytes rising).
• vmagent_remotewrite_packets_dropped_total increases if the queue overflows.
• Recent data lags behind in dashboards while vmagent drains its backlog.
• The receiving node logs too many outstanding insert requests or shows rate-limit rejections.

Common Root Causes

• Insert backpressure on the receiver — the target node/vminsert returns 429 because its insert concurrency is saturated (slow disk or too little capacity).
• Explicit rate limiting — a proxy, or vmagent’s own -remoteWrite.rateLimit, caps throughput and returns/produces 429s.
• Too few vmagent connections — -remoteWrite.queues is too low to push the sample volume, so blocks pile up and retries collide.
• Undersized receiver — the VictoriaMetrics cluster cannot keep up with total ingestion.
• Network or LB limits — an ingress/load balancer enforcing per-client request limits.
• Retry storm — aggressive retries amplify load right when the receiver is already overloaded.

Diagnostic Workflow

Confirm the direction of the problem from vmagent’s own metrics (port 8429):

curl -s http://localhost:8429/metrics | grep -E 'vmagent_remotewrite_(pending_data_bytes|packets_dropped_total|retries_count_total)'
curl -s http://localhost:8429/metrics | grep vmagent_remotewrite_requests_total

A rising vmagent_remotewrite_retries_count_total with 429 responses points at the receiver. Check the receiver’s insert saturation (single node 8428, vminsert 8480):

curl -s http://vm:8428/metrics | grep -E 'vm_concurrent_insert_(current|capacity)'
curl -s http://vm:8428/metrics | grep vm_rows_inserted_total

Look for an explicit rate limit on vmagent:

ps aux | grep vmagent | grep -oE '\-remoteWrite.(rateLimit|queues|maxDiskUsagePerURL)[^ ]*'

Chart ingestion vs capacity in vmui with MetricsQL:

sum(rate(vm_rows_inserted_total[5m]))
vm_concurrent_insert_current / vm_concurrent_insert_capacity

Example Root Cause Analysis

After adding 30 new nodes to a fleet, an SRE team saw every vmagent logging 429 Too Many Requests and vmagent_remotewrite_pending_data_bytes climbing toward the disk cap. On the receiver — a single-node VictoriaMetrics — vm_concurrent_insert_current was pinned at capacity and vm_slow_row_inserts_total was rising, meaning the disk could not keep up with the new sample rate. The 429s were downstream backpressure, not a rate-limit misconfiguration.

The immediate mitigation was to raise -remoteWrite.maxDiskUsagePerURL=4GB on the agents so bursts buffered safely and nothing was dropped. The durable fix was migrating the receiver to VictoriaMetrics cluster mode with three vmstorage nodes and two vminsert nodes, distributing the insert load. Once capacity matched ingestion, the 429s stopped and the agent queues drained to zero.

Prevention Best Practices

• Give vmagent a generous -remoteWrite.maxDiskUsagePerURL so transient 429s buffer to disk and retry instead of dropping data.
• Size receiver capacity (CPU, disk IOPS, or cluster nodes) to peak ingestion, not average.
• Tune -remoteWrite.queues to push enough concurrent blocks without overwhelming the receiver.
• Use -remoteWrite.rateLimit on agents deliberately to smooth spikes, and document any proxy-level limits.
• Alert on vmagent_remotewrite_pending_data_bytes and vmagent_remotewrite_packets_dropped_total so backlogs are caught before data loss.
• Reduce sample volume at the source with relabeling/aggregation (stream aggregation) where full resolution is not needed.

Quick Command Reference

# vmagent send-side health
curl -s http://localhost:8429/metrics | grep -E 'vmagent_remotewrite_(pending_data_bytes|packets_dropped_total|retries_count_total)'

# Receiver insert saturation
curl -s http://vm:8428/metrics | grep -E 'vm_concurrent_insert_(current|capacity)'

# Buffer bursts on disk instead of dropping
./vmagent -remoteWrite.url=http://vm:8428/api/v1/write \
  -remoteWrite.maxDiskUsagePerURL=4GB -remoteWrite.queues=8

# Optional deliberate rate smoothing
./vmagent -remoteWrite.rateLimit=200000

Conclusion

A 429 Too Many Requests on remote_write almost always means the receiver is applying backpressure because ingestion outpaces capacity. Use vmagent’s vmagent_remotewrite_* metrics to confirm the 429s and the receiver’s vm_concurrent_insert_current to prove saturation. Buffer with -remoteWrite.maxDiskUsagePerURL so no data is lost, then close the gap for good by adding IOPS, cores, or cluster nodes — or by shrinking the sample volume with stream aggregation.