Skip to content
DevOps AI ToolKit
Newsletter
All prompts
AI for Victoria Metrics Difficulty: Advanced ClaudeChatGPTCursor

Design VictoriaMetrics Stream Aggregation to Tame High-Cardinality Metrics

Design a vmagent or vmsingle stream aggregation config that pre-aggregates high-cardinality series at ingestion, collapsing labels while preserving counter and histogram semantics.

Target user
Platform and observability engineers running VictoriaMetrics who need to cut cardinality and storage cost without losing query fidelity.
Difficulty
Advanced
Tools
Claude, ChatGPT, Cursor

The prompt

You are a VictoriaMetrics ingestion engineer who has written production -streamAggr.config files, knows how `total`/`increase`/`histogram_bucket`/`quantiles` outputs differ, and understands how dedup_interval, staleness_interval, and interval interact with HA scrape pairs.

I will provide:
- The high-cardinality metrics I want to pre-aggregate (names, current label sets, and rough series counts from `/api/v1/status/tsdb` or vmui cardinality explorer)
- Where aggregation should run: vmagent (`-remoteWrite.streamAggr.config`) or vmsingle/vmstorage (`-streamAggr.config`), and whether HA scrape pairs feed it
- How these metrics are queried today (the MetricsQL / dashboard panels that consume them) and which labels are actually used in `by (...)` clauses
- Retention and cost pressure: storage growth, `vmstorage` RAM, or `slow inserts` I am trying to relieve
- Optionally: current scrape_interval, `-dedup.minScrapeInterval`, and any existing recording rules

Your job:

1. **Confirm what is safe to collapse** — cross-reference the labels the queries actually group by against the labels present on the raw series. Only labels never used in `by`/`without`/filters are candidates for dropping. Flag any label whose removal would break an existing panel or alert.

2. **Pick the correct outputs per metric type** — map each metric to the right `outputs`: counters → `total` or `increase` (explain the difference: `total` is monotonic-safe across resets and target churn, `increase` resets per interval); gauges → `avg`/`min`/`max`/`sum_samples`/`last`; histograms (`_bucket`) → `histogram_bucket`; summaries/quantiles → `quantiles(...)`. Call out any metric where `sum` would be wrong.

3. **Set the timing knobs deliberately** — recommend `interval` (aggregation window), `dedup_interval` (must equal scrape interval for HA pairs so replicas are deduplicated before aggregation), and `staleness_interval` (long enough to avoid spurious resets when targets disappear). Explain the failure mode of each if mis-set.

4. **Write the config block(s)** — produce the YAML for `-streamAggr.config` / `-remoteWrite.streamAggr.config` with `match`, `by` or `without`, `outputs`, `interval`, `dedup_interval`, and `keep_metric_names`/`output_relabel_configs` where naming matters. Comment every non-obvious line.

5. **Plan a safe rollout** — describe running with `keepInput: true` (or a second mirrored remote-write) so raw and aggregated series coexist, a MetricsQL query to compare `sum(rate(raw))` against the aggregated `total`, and the cutover step that drops the raw path once equivalence holds.

6. **List the observability of the aggregator itself** — the vmagent metrics to watch (`vm_streamaggr_*`, ignored samples, flush duration) and how to detect that an aggregation output is silently empty.

Output as: (a) label-collapse safety analysis, (b) per-metric output mapping table, (c) the annotated stream-aggregation YAML, (d) a raw-vs-aggregated equivalence check, (e) a staged rollout + rollback plan.

Bias toward correctness of counter/histogram semantics over maximum cardinality reduction; if collapsing a label would change query results or break an alert, say so explicitly and give me the tradeoff before recommending it.

Run this prompt with AI

Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.

Related prompts

More Victoria Metrics prompts & error guides

Browse every Victoria Metrics prompt and troubleshooting guide in one place.

Free download · 368-page PDF

Reading prompts? Get all 500 in one free PDF

500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.

  • 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
  • Instant PDF download — yours free, forever
  • Plus one practical AI-workflow email a week (no spam)

Single opt-in · unsubscribe anytime · no spam.