🚨 VictoriaMetrics Error: 'cannot keep up ... maxDiskUsagePerURL reached; dropping the oldest data' — Cause, Fix, and Troubleshooting Guide

Overview

vmagent buffers scraped and received data in a persistent on-disk queue before forwarding it to each remote storage URL. When the receiver cannot accept data fast enough, that queue grows. To avoid filling the disk, vmagent caps each URL’s queue at -remoteWrite.maxDiskUsagePerURL. Once the cap is hit, it starts discarding the oldest buffered data and logs:

the remote storage at "http://vminsert:8480/insert/0/prometheus/api/v1/write" cannot keep up with the data ingestion rate; -remoteWrite.maxDiskUsagePerURL=1073741824 bytes reached; dropping the oldest data

This is a backpressure signal: vmagent is producing (or has queued) more data than the remote endpoint is draining. The dropped data is gone. The real problem is almost always downstream — a slow or overloaded receiver, a saturated network, or a queue cap too small to ride out a spike. Raising the cap alone only buys time if the receiver can eventually catch up.

Symptoms

• vmagent logs cannot keep up with the data ingestion rate ... maxDiskUsagePerURL ... reached; dropping the oldest data.
• Gaps appear in graphs for data that should have been forwarded through this vmagent.
• The -remoteWrite.tmpDataPath directory sits pinned at (or near) the configured byte cap.
• vmagent CPU is fine but the remote endpoint (vminsert 8480, or a single-node 8428) is slow to respond or returns errors.
• The problem starts during a scrape spike, a receiver restart, or a network slowdown and clears once the receiver recovers.

Common Root Causes

• Remote storage down, slow, or overloaded — vminsert/vmstorage (or the single-node receiver) cannot accept writes quickly enough, so the queue backs up.
• -remoteWrite.maxDiskUsagePerURL too small — the cap cannot absorb the normal backlog during a brief outage or spike, so vmagent drops data prematurely.
• Network saturation — limited bandwidth or packet loss between vmagent and the remote URL throttles the drain rate below the ingestion rate.
• A sudden ingestion spike — a burst of new series or a scrape-interval change pushes throughput past what the receiver can sustain.

How to diagnose

Watch vmagent’s own metrics to see the queue filling and whether data is being dropped:

curl -s http://localhost:8429/metrics | grep -E \
  'vm_persistentqueue_bytes_pending|vmagent_remotewrite_(bytes_dropped_total|pending_data_bytes|requests_total)'

A rising vm_persistentqueue_bytes_pending toward the cap plus a growing vmagent_remotewrite_bytes_dropped_total confirms sustained backpressure. Check whether the receiver is the bottleneck by timing a direct write:

curl -s -o /dev/null -w 'code=%{http_code} time=%{time_total}s\n' \
  -X POST 'http://vminsert:8480/insert/0/prometheus/api/v1/write' --data-binary @/dev/null

Confirm the current disk cap and queue path in use:

ps aux | grep vmagent | grep -oE '\-remoteWrite.(maxDiskUsagePerURL|queues|tmpDataPath)[^ ]*'
df -h "$(ps aux | grep vmagent | grep -oE '\-remoteWrite.tmpDataPath=[^ ]*' | cut -d= -f2)"

If the receiver responds slowly or with 429/5xx, fix the receiver before touching vmagent limits.

Fixes

1. Raise -remoteWrite.maxDiskUsagePerURL. Give the queue enough headroom to survive a realistic receiver outage without dropping data — for example 16 GiB per URL:

./vmagent -remoteWrite.url=http://vminsert:8480/insert/0/prometheus/api/v1/write \
  -remoteWrite.maxDiskUsagePerURL=17179869184

Make sure -remoteWrite.tmpDataPath has at least this much free space per URL.

2. Add downstream capacity or fix the receiver. Scale vminsert/vmstorage (or the single-node receiver), address its CPU/IO/memory pressure, or resolve whatever is making it slow. This is the durable fix — the queue only backs up because the receiver cannot drain it.

3. Increase -remoteWrite.queues to drain faster. More concurrent sender queues push data out in parallel, which helps when the receiver has spare capacity but a single connection cannot saturate it:

./vmagent -remoteWrite.url=http://vminsert:8480/insert/0/prometheus/api/v1/write \
  -remoteWrite.queues=16

4. Ensure -remoteWrite.tmpDataPath has room. Point it at a volume with enough space for the raised cap across all URLs, and monitor its free space so the queue can actually reach its configured size.

5. Watch the queue and remote-write metrics. Alert on vm_persistentqueue_bytes_pending climbing toward the cap and on any increase in vmagent_remotewrite_bytes_dropped_total, so you catch backpressure before data loss instead of after.

What to watch out for

• Raising -remoteWrite.maxDiskUsagePerURL without fixing the receiver just delays the drops — the queue will still fill if the drain rate stays below the ingestion rate.
• Dropped data is not recoverable; the goal is to size the queue and receiver so drops never happen, not to recover them afterward.
• More -remoteWrite.queues helps only when the receiver has spare capacity — piling on queues against an overloaded receiver can make it worse.
• Confirm -remoteWrite.tmpDataPath free space actually exceeds the sum of per-URL caps; a small volume makes a large cap meaningless.
• A single vmagent can feed several URLs — the cap is per URL, so total disk need is the cap times the number of remote-write endpoints.

Related

• VictoriaMetrics Error Guide: remote write 429
• VictoriaMetrics Error Guide: remote write connection refused
• VictoriaMetrics Error Guide: no space left on device