🚨 VictoriaMetrics Error Guide: 'too many outstanding insert requests' — Widen the Insert Queue

Overview

VictoriaMetrics rejects an incoming write request when its insert concurrency limiter is saturated and the request has waited longer than the allowed queue duration. Clients like vmagent, Prometheus remote_write, or the Influx/Graphite ingestion paths receive an HTTP 503 with this body:

too many outstanding insert requests; try increasing -insert.maxQueueDuration=1m0s or -maxConcurrentInserts=8; see https://docs.victoriametrics.com/#troubleshooting

This is a backpressure signal: ingestion is arriving faster than the node (single-node victoria-metrics or vminsert in cluster mode) can persist it. The write is dropped rather than queued indefinitely, protecting the process from an unbounded memory blowup.

Symptoms

• Data collectors report HTTP 503 Service Unavailable on /api/v1/write, /write, or /insert/0/prometheus/api/v1/write.
• Gaps appear in dashboards during ingestion spikes; recent samples are missing.
• vmagent logs show remoteWrite retries and a growing persistent queue on disk.
• The metric vm_concurrent_insert_current sits at or near vm_concurrent_insert_capacity.
• CPU is not maxed out, but disk I/O wait is high on the data directory.

Common Root Causes

• Ingestion rate exceeds available insert concurrency — the default -maxConcurrentInserts (2× GOMAXPROCS) is too low for the sample rate.
• Slow storage — a spinning disk or throttled cloud volume makes each insert hold its slot longer, exhausting concurrency slots.
• vmstorage backpressure (cluster mode) — vminsert slots stay busy because one or more vmstorage nodes are slow or unavailable, so requests pile up.
• Large batch sizes — clients pushing very large remote_write batches occupy a slot for a long time each.
• Undersized node — too few CPU cores for the incoming samples-per-second rate.
• Queue duration too short — -insert.maxQueueDuration is set aggressively low, so brief spikes trip the limit instead of buffering.

Diagnostic Workflow

Confirm the limiter is the bottleneck by scraping the process metrics directly. On a single node this is port 8428; on vminsert it is 8480:

curl -s http://localhost:8428/metrics | grep -E 'vm_concurrent_insert_(current|capacity)'
curl -s http://localhost:8428/metrics | grep vm_rows_inserted_total

If vm_concurrent_insert_current equals vm_concurrent_insert_capacity, every slot is in use. Chart the ingestion rate in vmui with MetricsQL:

sum(rate(vm_rows_inserted_total[5m]))

Check whether storage latency is the real limit by looking at pending merges and I/O:

vm_pending_rows{type="storage"}
rate(vm_slow_row_inserts_total[5m])

Inspect the running insert flags to see current limits:

ps aux | grep -E 'victoria-metrics|vminsert' | grep -oE '\-(maxConcurrentInserts|insert.maxQueueDuration)[^ ]*'

In cluster mode, verify all vmstorage nodes are reachable from vminsert; a slow node throttles the whole insert path:

curl -s http://vmstorage-1:8482/metrics | grep vm_pending_rows

Example Root Cause Analysis

A single-node VictoriaMetrics on a 4-vCPU VM ingesting ~250k samples/s from 40 vmagents began returning 503 too many outstanding insert requests every weekday morning. vm_concurrent_insert_current pinned at the capacity of 8 (2× GOMAXPROCS on 4 cores) while rate(vm_slow_row_inserts_total[5m]) climbed. The data directory sat on a burstable gp2 EBS volume whose IOPS credits were exhausted by 09:00, so each insert held its slot far longer than normal, draining concurrency slots even though CPU was only 45% busy.

The immediate relief was raising -maxConcurrentInserts=16 and -insert.maxQueueDuration=1m to ride out spikes. The durable fix was migrating the data directory to a gp3 volume with provisioned IOPS, which cut vm_slow_row_inserts_total to near zero and let the default concurrency handle the load.

Prevention Best Practices

• Provision storage with adequate, non-burstable IOPS; slow disks are the most common hidden cause of insert backpressure.
• Size -maxConcurrentInserts to the node’s cores and observed sample rate rather than leaving the default on high-throughput nodes.
• Set -insert.maxQueueDuration (default 1m) high enough to absorb short spikes without dropping writes.
• On vmagent, configure -remoteWrite.maxDiskUsagePerURL so transient 503s buffer on disk and retry instead of losing data.
• Scale horizontally with cluster mode (vminsert/vmstorage) when a single node cannot keep up.
• Alert on vm_concurrent_insert_current / vm_concurrent_insert_capacity > 0.8 to catch saturation before writes fail.

Quick Command Reference

# Insert concurrency saturation
curl -s http://localhost:8428/metrics | grep -E 'vm_concurrent_insert_(current|capacity)'

# Ingestion rate (run in vmui / MetricsQL)
# sum(rate(vm_rows_inserted_total[5m]))

# Slow inserts caused by storage latency
curl -s http://localhost:8428/metrics | grep vm_slow_row_inserts_total

# Restart single node with wider limits
./victoria-metrics-prod -maxConcurrentInserts=16 -insert.maxQueueDuration=1m

# vmagent disk-backed retry buffer
./vmagent -remoteWrite.url=http://vm:8428/api/v1/write -remoteWrite.maxDiskUsagePerURL=2GB

Conclusion

too many outstanding insert requests is VictoriaMetrics telling you writes are arriving faster than it can safely persist them. Start by confirming whether the ceiling is insert concurrency (vm_concurrent_insert_current at capacity) or slow storage (vm_slow_row_inserts_total climbing). Raising -maxConcurrentInserts and -insert.maxQueueDuration buys headroom, but the lasting fix is almost always faster disks, more cores, or moving to cluster mode so ingestion scales with your sample rate instead of dropping it.