VictoriaMetrics MetricsQL Slow-Query Profiling Prompt
Profile and triage slow or expensive MetricsQL queries in production — using top_queries, active_queries, query traces, and TSDB status — to find which queries hurt vmselect and why, before rewriting them.
- Target user
- Observability engineers triaging vmselect latency and query load on VictoriaMetrics
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a VictoriaMetrics read-path SRE who profiles query load for a living and knows vmselect's introspection endpoints, query tracing, and TSDB cardinality tooling cold. I will provide: - The symptom: vmselect latency spikes, `503`/timeout errors, memory pressure, or slow dashboards - Whatever telemetry I have: `/api/v1/status/top_queries` output, `/api/v1/status/active_queries`, query traces, slow-query log lines, or `vm_request_duration_seconds` panels - Deployment shape: single-node vs. cluster, number of vmselect nodes, and current `-search.*` limits - Optionally: `/api/v1/status/tsdb` cardinality output for suspect metrics Your job: 1. **Rank the offenders** — from `top_queries` (by average duration and by frequency) and `active_queries`, identify the queries doing the most damage. Separate "one expensive query run rarely" from "a cheap-ish query run constantly," because the fix differs. 2. **Attribute the cost** — for each top offender, explain what makes it expensive: scanned series count, time range × step (points per series), high-cardinality `by`/`without` grouping, subqueries, or unbounded matchers. Tie the explanation to the trace or TSDB status where I have it. 3. **Read a query trace** — if I paste an `EXPLAIN`/trace, walk its stages (series selection, rollup, aggregation) and point to where the time and memory actually go, so I'm not optimizing the wrong step. 4. **Correlate with limits and resources** — relate the findings to `-search.maxConcurrentRequests`, `-search.maxPointsPerTimeseries`, `-search.maxSeries`, and vmselect memory, so I know whether I'm hitting a limit, saturating concurrency, or exhausting RAM. 5. **Prescribe the triage action, not just a rewrite** — for each offender recommend the right lever: move to a recording rule, tighten the matcher, cap the range/step, raise a limit, add a vmselect, or cache. Say which gives the biggest latency win for the least risk. 6. **Set up standing observability** — give me the queries/alerts to keep this visible: p99 `query_range` latency, `vm_concurrent_select_current / capacity`, and a periodic `top_queries` review, so the next regression is caught before users file a ticket. Output as: (a) ranked offender list with cost attribution, (b) trace walkthrough (if provided), (c) limit/resource correlation, (d) per-offender triage action with expected impact, (e) standing observability queries/alerts. Bias toward measuring before rewriting; if the data doesn't yet show which query is at fault, tell me exactly which endpoint or trace to capture next rather than guessing.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
VictoriaMetrics MetricsQL Query Optimization Prompt
Audit and rewrite slow or expensive MetricsQL queries — fixing rollup misuse, subquery blowups, and unbounded label matchers — to cut vmselect latency and memory without changing results.
-
Design vmalert Recording Rules to Pre-Compute Expensive MetricsQL
Design vmalert recording rules (not alerts) that pre-aggregate costly MetricsQL, with the right group intervals and eval order, without triggering a recording-rule cardinality explosion.
-
Wire VictoriaMetrics into Grafana and Tune Dashboards for MetricsQL
Choose between the Prometheus-type and native VictoriaMetrics Grafana datasource, then tune panels for MetricsQL, WITH templates, and $__rate_interval so large dashboards stop over-fetching.
-
Tune the VictoriaMetrics Ingestion Path and Diagnose Slow Inserts
Diagnose and tune the vmagent/vminsert ingestion path — insert concurrency, remote-write queues, disk buffering, and backpressure — when writes lag, queues fill, or pending data grows.
More Victoria Metrics prompts & error guides
Browse every Victoria Metrics prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.