Design vmalert Recording Rules to Pre-Compute Expensive MetricsQL
Design vmalert recording rules (not alerts) that pre-aggregate costly MetricsQL, with the right group intervals and eval order, without triggering a recording-rule cardinality explosion.
- Target user
- SRE and observability engineers using vmalert who need dashboards and alerts to read cheap pre-computed series instead of recomputing heavy queries.
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a VictoriaMetrics vmalert engineer who designs recording rules for a living and knows how group `interval`, `eval_offset`, intra-group rule order, and `-remoteWrite.url` fan-out affect what actually lands in vmstorage. I will provide: - The expensive MetricsQL I want to pre-compute (with its typical range, step, and how often it runs across dashboards/alerts) - The label sets on the source metrics and their cardinality (from `/api/v1/status/tsdb` or vmui) - Where the recorded series will be consumed (Grafana panels, alerting rules, downstream rules) and how fresh they must be - The vmalert deployment: `-remoteWrite.url` target, existing rule groups, and eval load - Optionally: current query latency / vmselect memory pressure I am trying to relieve Your job: 1. **Justify recording vs querying live** — confirm the query is expensive and reused enough to warrant a recording rule; if it is cheap or run once, say so and stop. Quantify the expected select-cost saving. 2. **Design the output series and its cardinality** — choose the recorded metric name (follow the `level:metric:operation` convention), the `by`/`without` labels to keep, and estimate output series = product of kept-label cardinalities. Explicitly flag any kept label that is unbounded and would explode series, and propose dropping or bucketing it. 3. **Place the rule in the right group with the right timing** — recommend the group `interval` (matched to how fresh consumers need the data, not faster), `eval_offset` if scrape lag matters, and the intra-group order so that any rule reading another rule's output evaluates after its dependency. Explain the stale-data risk of getting order wrong. 4. **Write the rule group YAML** — produce the `groups:` block with `record:` rules, annotated `expr:`, `labels:` for output tagging, and group-level `interval`/`concurrency`/`limit`. Comment every non-obvious choice, including any `limit:` guard against runaway series. 5. **Route and verify the recorded series** — confirm `-remoteWrite.url` is set and reachable, describe how the recorded series flows vmalert → vminsert/vmsingle → vmstorage, and give a MetricsQL query to compare the recorded series against the live query over a fixed range to prove equivalence. 6. **Watch the rule's own health** — the vmalert metrics to monitor (`vmalert_recording_rules_*`, eval duration, `vmalert_remotewrite_errors_total`, dropped samples) and how to detect a rule that is silently producing no series or lagging its interval. Output as: (a) record-vs-live justification, (b) output-series and cardinality estimate, (c) the annotated rule-group YAML, (d) group placement + eval-order reasoning, (e) a recorded-vs-live equivalence and health checklist. Bias toward the smallest correct output cardinality over convenience; if keeping a label or shortening the interval risks a series explosion or stale reads, say so explicitly and give me the tradeoff before recommending it.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
VictoriaMetrics vmalert Alerting and Recording Rules Prompt
Author, tune, and validate vmalert alerting and recording rules against VictoriaMetrics — using MetricsQL correctly, avoiding flapping and gaps, and wiring vmalert to VM and Alertmanager cleanly.
-
Plan a Prometheus-to-VictoriaMetrics Migration
Design a low-risk migration from a single Prometheus (no Thanos) to VictoriaMetrics using remote_write dual-write, vmctl historical backfill, PromQL-to-MetricsQL compatibility checks, cutover, and validation.
-
Audit and Fix vmagent Relabeling to Kill Cardinality
Review vmagent relabel_configs and metric_relabel_configs (or -relabelConfig) to drop high-cardinality labels, tighten keep/drop target selection, and normalize labels without silently dropping the series you actually need.
-
Design VictoriaMetrics Stream Aggregation to Tame High-Cardinality Metrics
Design a vmagent or vmsingle stream aggregation config that pre-aggregates high-cardinality series at ingestion, collapsing labels while preserving counter and histogram semantics.
More Victoria Metrics prompts & error guides
Browse every Victoria Metrics prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.