Design VictoriaMetrics Stream Aggregation to Tame High-Cardinality Metrics
Design a vmagent or vmsingle stream aggregation config that pre-aggregates high-cardinality series at ingestion, collapsing labels while preserving counter and histogram semantics.
- Target user
- Platform and observability engineers running VictoriaMetrics who need to cut cardinality and storage cost without losing query fidelity.
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a VictoriaMetrics ingestion engineer who has written production -streamAggr.config files, knows how `total`/`increase`/`histogram_bucket`/`quantiles` outputs differ, and understands how dedup_interval, staleness_interval, and interval interact with HA scrape pairs. I will provide: - The high-cardinality metrics I want to pre-aggregate (names, current label sets, and rough series counts from `/api/v1/status/tsdb` or vmui cardinality explorer) - Where aggregation should run: vmagent (`-remoteWrite.streamAggr.config`) or vmsingle/vmstorage (`-streamAggr.config`), and whether HA scrape pairs feed it - How these metrics are queried today (the MetricsQL / dashboard panels that consume them) and which labels are actually used in `by (...)` clauses - Retention and cost pressure: storage growth, `vmstorage` RAM, or `slow inserts` I am trying to relieve - Optionally: current scrape_interval, `-dedup.minScrapeInterval`, and any existing recording rules Your job: 1. **Confirm what is safe to collapse** — cross-reference the labels the queries actually group by against the labels present on the raw series. Only labels never used in `by`/`without`/filters are candidates for dropping. Flag any label whose removal would break an existing panel or alert. 2. **Pick the correct outputs per metric type** — map each metric to the right `outputs`: counters → `total` or `increase` (explain the difference: `total` is monotonic-safe across resets and target churn, `increase` resets per interval); gauges → `avg`/`min`/`max`/`sum_samples`/`last`; histograms (`_bucket`) → `histogram_bucket`; summaries/quantiles → `quantiles(...)`. Call out any metric where `sum` would be wrong. 3. **Set the timing knobs deliberately** — recommend `interval` (aggregation window), `dedup_interval` (must equal scrape interval for HA pairs so replicas are deduplicated before aggregation), and `staleness_interval` (long enough to avoid spurious resets when targets disappear). Explain the failure mode of each if mis-set. 4. **Write the config block(s)** — produce the YAML for `-streamAggr.config` / `-remoteWrite.streamAggr.config` with `match`, `by` or `without`, `outputs`, `interval`, `dedup_interval`, and `keep_metric_names`/`output_relabel_configs` where naming matters. Comment every non-obvious line. 5. **Plan a safe rollout** — describe running with `keepInput: true` (or a second mirrored remote-write) so raw and aggregated series coexist, a MetricsQL query to compare `sum(rate(raw))` against the aggregated `total`, and the cutover step that drops the raw path once equivalence holds. 6. **List the observability of the aggregator itself** — the vmagent metrics to watch (`vm_streamaggr_*`, ignored samples, flush duration) and how to detect that an aggregation output is silently empty. Output as: (a) label-collapse safety analysis, (b) per-metric output mapping table, (c) the annotated stream-aggregation YAML, (d) a raw-vs-aggregated equivalence check, (e) a staged rollout + rollback plan. Bias toward correctness of counter/histogram semantics over maximum cardinality reduction; if collapsing a label would change query results or break an alert, say so explicitly and give me the tradeoff before recommending it.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Audit and Fix vmagent Relabeling to Kill Cardinality
Review vmagent relabel_configs and metric_relabel_configs (or -relabelConfig) to drop high-cardinality labels, tighten keep/drop target selection, and normalize labels without silently dropping the series you actually need.
-
Wire VictoriaMetrics into Grafana and Tune Dashboards for MetricsQL
Choose between the Prometheus-type and native VictoriaMetrics Grafana datasource, then tune panels for MetricsQL, WITH templates, and $__rate_interval so large dashboards stop over-fetching.
-
VictoriaMetrics High Cardinality and Churn Control Prompt
Find and cut the labels driving cardinality explosion and high churn rate in VictoriaMetrics — using cardinality explorer, relabeling drops, and streaming aggregation — before they OOM vmstorage.
-
Design vmalert Recording Rules to Pre-Compute Expensive MetricsQL
Design vmalert recording rules (not alerts) that pre-aggregate costly MetricsQL, with the right group intervals and eval order, without triggering a recording-rule cardinality explosion.
More Victoria Metrics prompts & error guides
Browse every Victoria Metrics prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.