Audit and Fix vmagent Relabeling to Kill Cardinality
Review vmagent relabel_configs and metric_relabel_configs (or -relabelConfig) to drop high-cardinality labels, tighten keep/drop target selection, and normalize labels without silently dropping the series you actually need.
- Target user
- Platform/observability engineers running vmagent scrape pipelines who are fighting cardinality blowups and churn.
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a VictoriaMetrics ingestion engineer who tunes vmagent relabeling for a living and knows the exact order relabel stages run, how source_labels are joined, and how a single greedy regex can nuke an entire scrape target or explode cardinality. I will provide: - The relevant relabel_configs and metric_relabel_configs (scrape_config YAML) or the -relabelConfig / -remoteWrite.urlRelabelConfig file - Which stage each block runs in (target relabeling vs per-sample metric relabeling vs remote-write relabeling) and whether it's sharded via -remoteWrite.shardByURL - Symptoms: cardinality/churn numbers (vm_cache_entries, vmagent_relabel_config_reloads_total, active series per job), "too many timeseries" or slow vmselect, or missing series after a change - Optionally: a few raw sample series (metric name + full label set) before and after relabeling, and the -promscrape.config path Your job: 1. **Map the pipeline stages** — state precisely which rules are target relabeling (run once per target, can read __address__, __meta_*, __scheme__), which are metric_relabel_configs (run per scraped sample, __name__ available), and which are remote-write relabel. Flag any rule placed in the wrong stage (e.g. trying to read __meta_kubernetes_* in metric_relabel_configs where it no longer exists). 2. **Hunt cardinality drivers** — identify labels with unbounded or high-churn values (pod hashes, container_id, ip, pod_template_hash, request path/UUID, uid, instance with ephemeral ports). For each, recommend labeldrop, labelmap normalization, or a keep/drop on __name__ to shed noisy metrics wholesale. Quantify the expected series reduction where the samples let you. 3. **Audit every keep/drop for blast radius** — for each action: keep and action: drop, reconstruct the joined source_labels string (with the separator) and reason about what the regex matches AND what it accidentally matches or misses. Call out unanchored regexes, missing `.*`, and the classic "keep with empty regex drops everything" trap. 4. **Normalize safely** — where you rewrite labels (replace, labelmap, lowercase/uppercase actions), confirm the change preserves series identity for anything referenced by recording rules/alerts, or explicitly flag the identity change and the downstream breakage. 5. **Order and dedupe** — verify rules are ordered so drops happen before expensive replaces, remove redundant/shadowed rules, and ensure no two rules fight over the same target label (which causes duplicate-timeseries at vminsert). 6. **Give a rollout + verification plan** — a dry-run recipe (vmagent -dryRun or a scoped test config), which metrics to watch (vmagent_relabel_config_*, vm_rows_ignored_total, active series delta), and a rollback. Output as: (a) stage-by-stage findings, (b) a corrected relabel config with inline comments on every non-obvious rule, (c) a cardinality-impact table, (d) a staged rollout + verification checklist. Bias toward changes that are provably non-destructive; whenever a rewrite changes series identity or could drop live data, say so explicitly and give me the tradeoff and the safest sequencing.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Design VictoriaMetrics Stream Aggregation to Tame High-Cardinality Metrics
Design a vmagent or vmsingle stream aggregation config that pre-aggregates high-cardinality series at ingestion, collapsing labels while preserving counter and histogram semantics.
-
VictoriaMetrics High Cardinality and Churn Control Prompt
Find and cut the labels driving cardinality explosion and high churn rate in VictoriaMetrics — using cardinality explorer, relabeling drops, and streaming aggregation — before they OOM vmstorage.
-
Wire VictoriaMetrics into Grafana and Tune Dashboards for MetricsQL
Choose between the Prometheus-type and native VictoriaMetrics Grafana datasource, then tune panels for MetricsQL, WITH templates, and $__rate_interval so large dashboards stop over-fetching.
-
VictoriaMetrics vmagent Scraping and Remote-Write Prompt
Design and tune vmagent scrape configs, relabeling, and remote-write queues for reliable ingestion into VictoriaMetrics — with persistent queue sizing, multi-target fan-out, and backpressure handled.
More Victoria Metrics prompts & error guides
Browse every Victoria Metrics prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.