Skip to content
DevOps AI ToolKit
Newsletter
All prompts
AI for Victoria Metrics Difficulty: Advanced ClaudeChatGPTCursor

Audit and Fix vmagent Relabeling to Kill Cardinality

Review vmagent relabel_configs and metric_relabel_configs (or -relabelConfig) to drop high-cardinality labels, tighten keep/drop target selection, and normalize labels without silently dropping the series you actually need.

Target user
Platform/observability engineers running vmagent scrape pipelines who are fighting cardinality blowups and churn.
Difficulty
Advanced
Tools
Claude, ChatGPT, Cursor

The prompt

You are a VictoriaMetrics ingestion engineer who tunes vmagent relabeling for a living and knows the exact order relabel stages run, how source_labels are joined, and how a single greedy regex can nuke an entire scrape target or explode cardinality.

I will provide:
- The relevant relabel_configs and metric_relabel_configs (scrape_config YAML) or the -relabelConfig / -remoteWrite.urlRelabelConfig file
- Which stage each block runs in (target relabeling vs per-sample metric relabeling vs remote-write relabeling) and whether it's sharded via -remoteWrite.shardByURL
- Symptoms: cardinality/churn numbers (vm_cache_entries, vmagent_relabel_config_reloads_total, active series per job), "too many timeseries" or slow vmselect, or missing series after a change
- Optionally: a few raw sample series (metric name + full label set) before and after relabeling, and the -promscrape.config path

Your job:

1. **Map the pipeline stages** — state precisely which rules are target relabeling (run once per target, can read __address__, __meta_*, __scheme__), which are metric_relabel_configs (run per scraped sample, __name__ available), and which are remote-write relabel. Flag any rule placed in the wrong stage (e.g. trying to read __meta_kubernetes_* in metric_relabel_configs where it no longer exists).

2. **Hunt cardinality drivers** — identify labels with unbounded or high-churn values (pod hashes, container_id, ip, pod_template_hash, request path/UUID, uid, instance with ephemeral ports). For each, recommend labeldrop, labelmap normalization, or a keep/drop on __name__ to shed noisy metrics wholesale. Quantify the expected series reduction where the samples let you.

3. **Audit every keep/drop for blast radius** — for each action: keep and action: drop, reconstruct the joined source_labels string (with the separator) and reason about what the regex matches AND what it accidentally matches or misses. Call out unanchored regexes, missing `.*`, and the classic "keep with empty regex drops everything" trap.

4. **Normalize safely** — where you rewrite labels (replace, labelmap, lowercase/uppercase actions), confirm the change preserves series identity for anything referenced by recording rules/alerts, or explicitly flag the identity change and the downstream breakage.

5. **Order and dedupe** — verify rules are ordered so drops happen before expensive replaces, remove redundant/shadowed rules, and ensure no two rules fight over the same target label (which causes duplicate-timeseries at vminsert).

6. **Give a rollout + verification plan** — a dry-run recipe (vmagent -dryRun or a scoped test config), which metrics to watch (vmagent_relabel_config_*, vm_rows_ignored_total, active series delta), and a rollback.

Output as: (a) stage-by-stage findings, (b) a corrected relabel config with inline comments on every non-obvious rule, (c) a cardinality-impact table, (d) a staged rollout + verification checklist.

Bias toward changes that are provably non-destructive; whenever a rewrite changes series identity or could drop live data, say so explicitly and give me the tradeoff and the safest sequencing.

Run this prompt with AI

Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.

Related prompts

More Victoria Metrics prompts & error guides

Browse every Victoria Metrics prompt and troubleshooting guide in one place.

Free download · 368-page PDF

Reading prompts? Get all 500 in one free PDF

500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.

  • 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
  • Instant PDF download — yours free, forever
  • Plus one practical AI-workflow email a week (no spam)

Single opt-in · unsubscribe anytime · no spam.