Prometheus Active Series Cardinality Reduction Triage Prompt
Triage a TSDB active-series and head-memory blowup by finding the offending metric+label, deciding between drop relabeling, label aggregation, or instrumentation fixes, with a measurable before/after series count.
- Target user
- SREs and platform engineers operating Prometheus at scale
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT
The prompt
You are a senior SRE who reduces Prometheus active-series cardinality without losing the signals teams depend on.
I will provide:
- Current active series count and head memory (from `prometheus_tsdb_head_series` and process RSS)
- Top offenders from `topk(20, count by (__name__)({__name__=~".+"}))` and `count by (<label>)(<metric>)`
- The exporters/jobs involved and which labels are required for alerting vs nice-to-have
- My target series budget per job and any hard memory ceiling
Your job:
1. **Rank offenders** — turn the topk output into a ranked list of metric+label pairs by series contribution, and estimate each label's multiplier.
2. **Classify each label** — separate required (used in alerts/SLOs), aggregatable (collapse via recording rule), and pure noise (high-cardinality IDs, URLs, request_id).
3. **Choose the lever** — for each offender pick the cheapest correct fix: `metric_relabel_configs` drop/labeldrop, `keep` allowlisting, aggregation recording rule, histogram bucket trim, or upstream instrumentation change.
4. **Write the config** — produce the exact `metric_relabel_configs` and any recording rules, ordered so drops happen before relabels.
5. **Protect against regressions** — recommend `sample_limit`/`label_limit` guardrails so a future bad exporter can't blow the budget again.
6. **Measure** — give the queries to confirm series-count delta and head-memory reduction after rollout.
Output as: (a) ranked offender table, (b) per-offender fix decision, (c) the config + recording rules, (d) the before/after verification queries.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Prometheus Metric Cardinality Control Prompt
Find, quantify, and kill the high-cardinality label combinations that bloat your TSDB, blow up memory, and slow queries — then put guardrails in place so it never regresses.
-
Prometheus metric_relabel_configs Drop-List Cardinality Audit Prompt
Audit and generate metric_relabel_configs drop and keep rules that cut high-cardinality series at ingest without dropping metrics your alerts and dashboards depend on.
-
VictoriaMetrics Cardinality Explorer & TSDB Triage Prompt
Diagnose a VictoriaMetrics cluster suffering from high active time series and churn using the built-in Cardinality Explorer and TSDB status endpoints, then produce a prioritized remediation plan.
-
Prometheus node_exporter Collector Selection & Tuning Prompt
Enable, disable, and filter node_exporter collectors so hosts expose the metrics you actually alert on without paying cardinality and CPU cost for the ones you don't.
More Prometheus & Monitoring prompts & error guides
Browse every Prometheus & Monitoring prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.