AI for Victoria Metrics
Operate and tune VictoriaMetrics with AI — MetricsQL, vmagent and vmstorage, high-cardinality ingestion, downsampling, retention, and cluster scaling.
21 copy-paste prompts · 50 in-depth guides Jump to prompts Jump to guides
Prompts
- Advanced
VictoriaMetrics Downsampling ROI and Storage Cost Forecasting Prompt
Model the dollar and disk impact of downsampling and retention changes in VictoriaMetrics — projecting storage growth, per-tier savings, and payback — so cost decisions are made on arithmetic instead of vibes.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
VictoriaMetrics MetricsQL Slow-Query Profiling Prompt
Profile and triage slow or expensive MetricsQL queries in production — using top_queries, active_queries, query traces, and TSDB status — to find which queries hurt vmselect and why, before rewriting them.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
VictoriaMetrics vmagent Scrape Target Debugging Prompt
Systematically diagnose down, missing, or flapping scrape targets in vmagent — walking service discovery, relabeling drops, and scrape failures — so metrics stop silently disappearing from VictoriaMetrics.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
VictoriaMetrics Capacity Planning and Resource Sizing Prompt
Build a capacity model for VictoriaMetrics — deriving CPU, RAM, disk, and node counts from ingestion rate, active series, churn, and retention, with headroom for query load and growth.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
VictoriaMetrics Single-Node vs Cluster Scaling Prompt
Decide when to move from single-node VictoriaMetrics to the cluster (vminsert/vmselect/vmstorage) topology, size the storage tier, and scale it out safely with replication and rerouting understood.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
VictoriaMetrics Deduplication and HA Pairs Prompt
Configure deduplication for highly-available Prometheus/vmagent replica pairs writing to VictoriaMetrics — setting the right dedup interval and label strategy so redundant samples collapse cleanly without gaps or double-counting.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
VictoriaMetrics Downsampling and Retention Policy Prompt
Design tiered retention and downsampling in VictoriaMetrics Enterprise — mapping metric classes to resolutions and retention windows to cut disk and speed long-range queries without losing what matters.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
Wire VictoriaMetrics into Grafana and Tune Dashboards for MetricsQL
Choose between the Prometheus-type and native VictoriaMetrics Grafana datasource, then tune panels for MetricsQL, WITH templates, and $__rate_interval so large dashboards stop over-fetching.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
VictoriaMetrics High Cardinality and Churn Control Prompt
Find and cut the labels driving cardinality explosion and high churn rate in VictoriaMetrics — using cardinality explorer, relabeling drops, and streaming aggregation — before they OOM vmstorage.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
Tune the VictoriaMetrics Ingestion Path and Diagnose Slow Inserts
Diagnose and tune the vmagent/vminsert ingestion path — insert concurrency, remote-write queues, disk buffering, and backpressure — when writes lag, queues fill, or pending data grows.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
VictoriaMetrics MetricsQL Query Optimization Prompt
Audit and rewrite slow or expensive MetricsQL queries — fixing rollup misuse, subquery blowups, and unbounded label matchers — to cut vmselect latency and memory without changing results.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
Design Multitenancy and Tenant Isolation on VM Cluster
Design multitenancy on a VictoriaMetrics cluster — accountID:projectID URL routing through vminsert/vmselect, per-tenant limits and rate controls, isolation guarantees, and cost attribution.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
Plan a Prometheus-to-VictoriaMetrics Migration
Design a low-risk migration from a single Prometheus (no Thanos) to VictoriaMetrics using remote_write dual-write, vmctl historical backfill, PromQL-to-MetricsQL compatibility checks, cutover, and validation.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
Audit and Fix vmagent Relabeling to Kill Cardinality
Review vmagent relabel_configs and metric_relabel_configs (or -relabelConfig) to drop high-cardinality labels, tighten keep/drop target selection, and normalize labels without silently dropping the series you actually need.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
Design VictoriaMetrics Stream Aggregation to Tame High-Cardinality Metrics
Design a vmagent or vmsingle stream aggregation config that pre-aggregates high-cardinality series at ingestion, collapsing labels while preserving counter and histogram semantics.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
VictoriaMetrics Migration from Thanos Prompt
Plan a migration from a Thanos stack (sidecar, store gateway, compactor, object storage) to VictoriaMetrics — consolidating the moving parts, backfilling historical blocks, and preserving global query and long-term storage.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
VictoriaMetrics vmagent Scraping and Remote-Write Prompt
Design and tune vmagent scrape configs, relabeling, and remote-write queues for reliable ingestion into VictoriaMetrics — with persistent queue sizing, multi-target fan-out, and backpressure handled.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
VictoriaMetrics vmalert Alerting and Recording Rules Prompt
Author, tune, and validate vmalert alerting and recording rules against VictoriaMetrics — using MetricsQL correctly, avoiding flapping and gaps, and wiring vmalert to VM and Alertmanager cleanly.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
Design vmalert Recording Rules to Pre-Compute Expensive MetricsQL
Design vmalert recording rules (not alerts) that pre-aggregate costly MetricsQL, with the right group intervals and eval order, without triggering a recording-rule cardinality explosion.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
Configure vmauth as an Auth and Routing Gateway
Configure vmauth in front of a VictoriaMetrics cluster — url_map routing, bearer/basic auth per user, per-user rate limiting, load balancing across vmselect/vminsert, and TLS termination.
- Claude
- ChatGPT
- Cursor
Open prompt - Advanced
Design a vmbackup/vmrestore Disaster-Recovery Strategy
Design an end-to-end vmbackup, vmbackupmanager, and vmrestore DR plan with snapshots, object-storage layout, incremental backups, and rehearsed restore drills mapped to real RTO/RPO targets.
- Claude
- ChatGPT
- Cursor
Open prompt
Guides
- · 7 min read
VictoriaMetrics Error: 'bind: address already in use' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'cannot start the server at ":8428": bind: address already in use': find the port holder, stop stale processes, or move -httpListenAddr.
Read guide - · 8 min read
VictoriaMetrics Error: 'cannot create snapshot: invalid cross-device link' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'cannot create snapshot ... invalid cross-device link': keep snapshots on the -storageDataPath filesystem and set -snapshotAuthKey.
Read guide - · 7 min read
VictoriaMetrics Error: 'cannot decode snappy-encoded remote_write request body: snappy: corrupt input' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'cannot decode snappy-encoded remote_write request body: snappy: corrupt input': send real snappy and stop proxies re-encoding it.
Read guide - · 8 min read
VictoriaMetrics Error: 'cannot load -promscrape.config' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'cannot load -promscrape.config ...: did not find expected key': validate scrape YAML, dry-run, find the bad field, reload.
Read guide - · 7 min read
VictoriaMetrics Error: 'cannot load relabel configs ... unsupported relabel action' — Cause, Fix, and Troubleshooting Guide
Fix vmagent 'cannot load relabel configs ... unsupported relabel action': use supported actions, set source_labels, fix regex, dry-run first.
Read guide - · 8 min read
VictoriaMetrics Error: 'cannot open file ... no such file or directory' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'cannot open file ... no such file or directory': recover incomplete parts after an unclean shutdown, check disk, restore backup.
Read guide - · 7 min read
VictoriaMetrics Error: 'cannot unmarshal Graphite plaintext line ... unsupported format' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'cannot unmarshal Graphite plaintext line ...: cannot parse timestamp': send metric value timestamp with numeric epoch fields.
Read guide - · 8 min read
VictoriaMetrics Error: 'cannot parse labels' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'cannot parse labels ...: missing quotes for label value': quote label values, fix exporter text, post to the right endpoint.
Read guide - · 7 min read
VictoriaMetrics Error: 'cannot parse metric value' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'cannot parse metric value ... cannot parse "1,25" as float': emit dot-decimal floats, fix the exporter, validate /metrics.
Read guide - · 7 min read
VictoriaMetrics Error: 'cannot parse timestamp' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'cannot parse timestamp ...: too big timestamp; probably wrong precision': send the right epoch precision on ingestion.
Read guide - · 7 min read
VictoriaMetrics Error: 'cannot unmarshal labels ... the data is corrupted' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'cannot unmarshal labels ... probably the data is corrupted': resolve version skew, wrong ingest paths, and mangling proxies.
Read guide - · 8 min read
VictoriaMetrics Error: 'duplicate time series on the right side of the operation' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'duplicate time series on the right side of the operation': add on()/ignoring(), use group_left, or aggregate duplicates away.
Read guide - · 8 min read
VictoriaMetrics Error: 'cannot discover kubernetes targets ... 403 Forbidden' — Cause, Fix, and Troubleshooting Guide
Fix vmagent 'cannot discover kubernetes targets ... 403 Forbidden; ServiceAccount missing list/watch RBAC': grant RBAC, mount the SA token, verify access.
Read guide - · 8 min read
VictoriaMetrics Error: 'the requested time range duration exceeds -search.maxExportDuration' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'the requested time range duration exceeds -search.maxExportDuration=720h0m0s': chunk the range, use native export or vmbackup.
Read guide - · 8 min read
VictoriaMetrics Error: 'dropping label ... too long value; -maxLabelValueLen=1024 exceeded' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'dropping label ... too long value; -maxLabelValueLen=1024 exceeded': truncate labels via relabeling, fix instrumentation, raise limit.
Read guide - · 8 min read
VictoriaMetrics Error: 'the query requires more than -search.maxMemoryPerQuery bytes of memory' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'the query requires more than -search.maxMemoryPerQuery bytes of memory': narrow filters, split the query, or raise the budget.
Read guide - · 7 min read
VictoriaMetrics Error: 'too long query' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'too long query; the maximum allowed query length is -search.maxQueryLen=16384 bytes': collapse or chains to regex, raise the limit.
Read guide - · 8 min read
VictoriaMetrics Error: 'cannot select more than -search.maxSamplesPerQuery samples' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'cannot select more than -search.maxSamplesPerQuery samples': narrow label filters, shrink the range, or raise with RAM headroom.
Read guide - · 8 min read
VictoriaMetrics Error: '-search.maxTagValues=100000 label values found' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'error when searching for label values: -search.maxTagValues=100000 label values found': scope matchers, chain vars, cut cardinality.
Read guide - · 8 min read
VictoriaMetrics Error: 'cannot keep up ... maxDiskUsagePerURL reached; dropping the oldest data' — Cause, Fix, and Troubleshooting Guide
Fix vmagent '-remoteWrite.maxDiskUsagePerURL reached; dropping the oldest data': add receiver capacity, raise the disk cap, and drain the queue faster.
Read guide - · 8 min read
VictoriaMetrics Error: 'x509: certificate signed by unknown authority' — Cause, Fix, and Troubleshooting Guide
Fix vmagent 'cannot send a block to remote storage ... x509: certificate signed by unknown authority': trust the CA, set tlsServerName, renew certs safely.
Read guide - · 8 min read
VictoriaMetrics Error: 'unexpected status code received after sending a block ... 400' — Cause, Fix, and Troubleshooting Guide
Fix vmagent 'unexpected status code received after sending a block ... 400; cannot unmarshal MetricRow': read the body, fix the URL/tenant path, align versions.
Read guide - · 7 min read
VictoriaMetrics Error: 'missing -remoteWrite.url command-line flag' — Cause, Fix, and Troubleshooting Guide
Fix vmagent 'missing -remoteWrite.url command-line flag; it must contain the url of the remote storage': set the flag, verify env expansion, fan out backends.
Read guide - · 7 min read
VictoriaMetrics Error: 'mkdir /victoria-metrics-data: permission denied' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'cannot create directory ... permission denied' on -storageDataPath: fix volume ownership, SELinux relabel, and Kubernetes fsGroup.
Read guide - · 8 min read
VictoriaMetrics Error: 'cannot load stream aggregation config: unexpected output' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'cannot load stream aggregation config ... unexpected output': use supported outputs, valid interval, correct match[] selector, then reload.
Read guide - · 8 min read
VictoriaMetrics Error: 'unexpected end of stream' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'unexpected end of stream; the connection was probably closed while transferring data': raise LB timeouts, tune vmagent queues.
Read guide - · 8 min read
VictoriaMetrics Error: 'unsupported path requested' — Cause, Fix, and Troubleshooting Guide
Fix VictoriaMetrics 'unsupported path requested: /api/v1/write': target the right component and port with the correct /insert or /select tenant path.
Read guide - · 8 min read
VictoriaMetrics Error: 'cannot execute rule ... unexpected response from datasource' — Cause, Fix, and Troubleshooting Guide
Fix vmalert 'unexpected response from datasource ... status code 422; cannot parse query': fix -datasource.url and tenant prefix, validate rules, add auth.
Read guide - · 7 min read
VictoriaMetrics Error: 'cannot authorize request: missing Authorization request header' — Cause, Fix, and Troubleshooting Guide
Fix vmauth 'cannot authorize request: missing Authorization request header' HTTP 401: send valid credentials, define the user in -auth.config, reload vmauth.
Read guide - · 8 min read
VictoriaMetrics Error Guide: 'the number of matching time series exceeds -search.maxSeries' — Bound the Match Set
Fix VictoriaMetrics 'the number of matching time series exceeds -search.maxSeries': tighten label matchers, cut cardinality, and raise the series limit safely on vmselect.
Read guide - · 9 min read
VictoriaMetrics Error Guide: 'no space left on device' — Reclaim and Cap Disk
Fix VictoriaMetrics 'no space left on device': free vmstorage disk, cap the vmagent persistent queue, and cut retention/cardinality before writes are lost.
Read guide - · 9 min read
VictoriaMetrics Error Guide: 'the response from ... exceeds -promscrape.maxScrapeSize' — Right-Size the Scrape
Fix vmagent 'the response from ... exceeds -promscrape.maxScrapeSize': trim exporter output, raise the scrape size limit, and cut cardinality before targets stop reporting.
Read guide - · 8 min read
VictoriaMetrics Error Guide: 'cannot obtain lock on file flock.lock' — Ensure a Single Owner of the Data Directory
Fix 'cannot obtain lock on file /victoria-metrics-data/flock.lock' in VictoriaMetrics: stop the duplicate process, clear the stale lock, and give the data dir one owner.
Read guide - · 8 min read
VictoriaMetrics Error Guide: 'cannot parse MetricsQL' — Fix Broken Query Syntax
Fix VictoriaMetrics 'cannot parse MetricsQL': read the position marker, close brackets and quotes, correct label matchers and durations, and validate MetricsQL vs PromQL differences in vmui.
Read guide - · 9 min read
VictoriaMetrics Error Guide: vmagent 'connection reset by peer' to vmstorage — Restore the Ingestion Path
Fix VictoriaMetrics 'connection reset by peer' between vminsert/vmagent and vmstorage: check the 8400/8401 ports, OOM restarts, network policies, MTU, and version skew across cluster nodes.
Read guide - · 9 min read
VictoriaMetrics Error Guide: 'cannot execute query: context deadline exceeded' — Raise Timeout or Optimize the Query
Fix 'cannot execute query: context deadline exceeded' in VictoriaMetrics: raise -search.maxQueryDuration, optimize heavy MetricsQL, and scale vmstorage to finish queries in time.
Read guide - · 9 min read
VictoriaMetrics Error Guide: 'too high churn rate' — Stop Time Series Churn
Fix VictoriaMetrics 'too high churn rate' warnings: find labels that change every scrape, drop volatile labels with relabeling, and cut cardinality driving slow queries and memory growth.
Read guide - · 8 min read
VictoriaMetrics Error Guide: 'labels count exceeds the limit' — Trim Per-Series Labels
Fix VictoriaMetrics 'labels count exceeds -maxLabelsPerTimeseries': find the over-labeled metric, drop excess labels with relabeling at ingestion, and raise the limit only when labels are legitimate.
Read guide - · 9 min read
VictoriaMetrics Error Guide: 'cannot handle more than -search.maxConcurrentRequests concurrent requests' — Tame Query Load
Fix VictoriaMetrics 'cannot handle more than -search.maxConcurrentRequests concurrent requests': tune the query concurrency limit and queue duration, cut heavy queries, and scale vmselect.
Read guide - · 9 min read
VictoriaMetrics Error Guide: 'the number of unique series exceeds -search.maxUniqueTimeseries' — Cut Query Cardinality
Fix VictoriaMetrics 'the number of unique series exceeds -search.maxUniqueTimeseries': narrow label matchers, drop high-cardinality labels, use recording rules, and raise the limit safely.
Read guide - · 8 min read
VictoriaMetrics Error Guide: 'missing `accountID` in the request path' — Fix Multitenant URLs
Fix VictoriaMetrics cluster 'missing accountID in the request path': use the correct /insert/<accountID>/ and /select/<accountID>/ URLs with a numeric accountID:projectID tenant.
Read guide - · 9 min read
VictoriaMetrics Error Guide: 'Out of memory: Killed process 1234 (victoria-metrics)' — Cap Memory and Concurrency
Fix VictoriaMetrics OOM kills: tune -memory.allowedPercent, cap concurrent queries, cut cardinality, and scale vmselect/vmstorage RAM before the kernel kills the process again.
Read guide - · 9 min read
VictoriaMetrics Error Guide: 'insufficient free disk space; switching to read-only mode' — Free or Grow the Storage Disk
Fix 'insufficient free disk space; switching to read-only mode' in VictoriaMetrics: free or grow the disk, tune -storage.minFreeDiskSpaceBytes, retention, and alert on free space.
Read guide - · 9 min read
VictoriaMetrics Error Guide: remote_write '429 Too Many Requests' — Relieve Ingestion Throttling
Fix VictoriaMetrics remote_write '429 Too Many Requests' from vmagent: understand insert backpressure and rate limits, buffer with disk queues, tune concurrency, and scale ingestion capacity.
Read guide - · 9 min read
VictoriaMetrics Error Guide: 'connection refused' on vmagent remoteWrite — Restore the vminsert Path
Fix vmagent 'dial tcp: connect: connection refused' to vminsert:8480: check the service is up, verify port/DNS/firewall, and drain the on-disk buffer safely.
Read guide - · 9 min read
VictoriaMetrics Error Guide: 'too many open files' — Raise File Descriptor Limits
Fix VictoriaMetrics 'too many open files': raise the systemd LimitNOFILE and ulimit -n, understand mmap and part file usage, reduce cardinality, and monitor open descriptors before they run out.
Read guide - · 9 min read
VictoriaMetrics Error Guide: 'too many outstanding insert requests' — Widen the Insert Queue
Fix 'too many outstanding insert requests' in VictoriaMetrics: tune -maxConcurrentInserts and -insert.maxQueueDuration, scale ingestion, and relieve slow disk and vmstorage backpressure.
Read guide - · 8 min read
VictoriaMetrics Error Guide: 'too many points for the given step and time range' — Increase Step or Raise the Points Limit
Fix 'too many points for the given step and time range' in VictoriaMetrics: increase the query step, narrow the time range, or raise -search.maxPointsPerTimeseries safely.
Read guide - · 9 min read
VictoriaMetrics Error Guide: 'dropping row with too old timestamp' — Backfill and Retention Fix
Fix 'dropping row with too old timestamp' in VictoriaMetrics: backfill correctly with vmctl, extend -retentionPeriod, tune dedup, and eliminate clock skew with NTP.
Read guide - · 9 min read
VictoriaMetrics Error Guide: 'cannot execute query on all the vmstorage nodes; the following nodes are unavailable' — Restore vmselect-to-vmstorage Connectivity
Fix 'cannot execute query on all the vmstorage nodes; nodes are unavailable' in VictoriaMetrics: check vmstorage health, port 8401, network, and -storageNode flags.
Read guide