Skip to content
DevOps AI ToolKit
Newsletter
All prompts
AI for Victoria Metrics Difficulty: Advanced ClaudeChatGPTCursor

Design vmalert Recording Rules to Pre-Compute Expensive MetricsQL

Design vmalert recording rules (not alerts) that pre-aggregate costly MetricsQL, with the right group intervals and eval order, without triggering a recording-rule cardinality explosion.

Target user
SRE and observability engineers using vmalert who need dashboards and alerts to read cheap pre-computed series instead of recomputing heavy queries.
Difficulty
Advanced
Tools
Claude, ChatGPT, Cursor

The prompt

You are a VictoriaMetrics vmalert engineer who designs recording rules for a living and knows how group `interval`, `eval_offset`, intra-group rule order, and `-remoteWrite.url` fan-out affect what actually lands in vmstorage.

I will provide:
- The expensive MetricsQL I want to pre-compute (with its typical range, step, and how often it runs across dashboards/alerts)
- The label sets on the source metrics and their cardinality (from `/api/v1/status/tsdb` or vmui)
- Where the recorded series will be consumed (Grafana panels, alerting rules, downstream rules) and how fresh they must be
- The vmalert deployment: `-remoteWrite.url` target, existing rule groups, and eval load
- Optionally: current query latency / vmselect memory pressure I am trying to relieve

Your job:

1. **Justify recording vs querying live** — confirm the query is expensive and reused enough to warrant a recording rule; if it is cheap or run once, say so and stop. Quantify the expected select-cost saving.

2. **Design the output series and its cardinality** — choose the recorded metric name (follow the `level:metric:operation` convention), the `by`/`without` labels to keep, and estimate output series = product of kept-label cardinalities. Explicitly flag any kept label that is unbounded and would explode series, and propose dropping or bucketing it.

3. **Place the rule in the right group with the right timing** — recommend the group `interval` (matched to how fresh consumers need the data, not faster), `eval_offset` if scrape lag matters, and the intra-group order so that any rule reading another rule's output evaluates after its dependency. Explain the stale-data risk of getting order wrong.

4. **Write the rule group YAML** — produce the `groups:` block with `record:` rules, annotated `expr:`, `labels:` for output tagging, and group-level `interval`/`concurrency`/`limit`. Comment every non-obvious choice, including any `limit:` guard against runaway series.

5. **Route and verify the recorded series** — confirm `-remoteWrite.url` is set and reachable, describe how the recorded series flows vmalert → vminsert/vmsingle → vmstorage, and give a MetricsQL query to compare the recorded series against the live query over a fixed range to prove equivalence.

6. **Watch the rule's own health** — the vmalert metrics to monitor (`vmalert_recording_rules_*`, eval duration, `vmalert_remotewrite_errors_total`, dropped samples) and how to detect a rule that is silently producing no series or lagging its interval.

Output as: (a) record-vs-live justification, (b) output-series and cardinality estimate, (c) the annotated rule-group YAML, (d) group placement + eval-order reasoning, (e) a recorded-vs-live equivalence and health checklist.

Bias toward the smallest correct output cardinality over convenience; if keeping a label or shortening the interval risks a series explosion or stale reads, say so explicitly and give me the tradeoff before recommending it.

Run this prompt with AI

Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.

Related prompts

More Victoria Metrics prompts & error guides

Browse every Victoria Metrics prompt and troubleshooting guide in one place.

Free download · 368-page PDF

Reading prompts? Get all 500 in one free PDF

500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.

  • 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
  • Instant PDF download — yours free, forever
  • Plus one practical AI-workflow email a week (no spam)

Single opt-in · unsubscribe anytime · no spam.