🚨 VictoriaMetrics Error Guide: 'too high churn rate' — Stop Time Series Churn

Overview

VictoriaMetrics logs a churn-rate warning when the number of brand-new time series it creates per unit time rivals the number of series being actively updated. The log line looks like:

the churn rate is high: 1250000 new time series created over the last 24h; this may lead to high memory usage and slow queries; see https://docs.victoriametrics.com/faq/#what-is-high-churn-rate

Churn means old series stop receiving samples and new series with different label sets take their place. High churn inflates the index, bloats memory, slows queries, and is the leading cause of runaway cardinality in VictoriaMetrics.

Symptoms

• The victoria-metrics/vmstorage log repeatedly warns about high churn rate.
• vm_new_timeseries_created_total climbs steeply relative to active series.
• Memory usage grows over hours or days with no rise in real workload.
• Range queries get slower over time even though instantaneous cardinality looks bounded.
• Disk usage for the index (indexdb) grows disproportionately to raw sample data.

Common Root Causes

• Volatile labels — a label whose value changes every scrape (deployment hash, pod name with random suffix, timestamp, build ID) creates a new series each time.
• Ephemeral workloads — Kubernetes pods, CronJobs, and CI runners that come and go rapidly, each producing uniquely-labeled series.
• Unbounded identifiers in labels — request_id, trace_id, or session_id used as labels.
• Frequent relabeling changes — target relabeling that reshuffles label values on every reload.
• Auto-generated metric names — instrumenting with dynamic metric names rather than a fixed name plus labels.

Diagnostic Workflow

Measure the churn directly with MetricsQL in vmui:

sum(rate(vm_new_timeseries_created_total[5m]))

Compare that against active series to gauge the ratio:

sum(rate(vm_new_timeseries_created_total[1h])) / vm_cache_entries{type="storage/hour_metric_ids"}

Use the cardinality explorer in vmui (Cardinality tab) or the TSDB status API to find which metric and label churns:

curl -s 'http://localhost:8428/api/v1/status/tsdb' | head

Look at the seriesCountByLabelValuePair and labelValueCountByLabelName sections — a label with a value count in the millions that keeps climbing is your churning label. Confirm a candidate label’s value count over time:

count(count by (pod) (up{job="my-app"}))

If that value climbs steadily while the number of running pods is stable, the pod label (with random suffixes) is churning.

Example Root Cause Analysis

A platform team saw victoria-metrics memory grow from 6 GB to 22 GB over a week alongside repeated “the churn rate is high” logs. sum(rate(vm_new_timeseries_created_total[5m])) was ~200/s despite a stable service count. The cardinality explorer flagged a metric app_build_info carrying a git_commit label plus a pod label with Kubernetes-generated random suffixes. Every deploy and every pod restart minted a fresh series, and CI deployed dozens of times a day.

The fix was metric_relabel_configs in vmagent to drop the git_commit label from all but a single low-frequency build_info metric, and to rewrite the pod label to the owning Deployment name using regex relabeling. New series creation fell by 95%, memory stabilized, and range queries sped up because indexdb stopped ballooning.

Prevention Best Practices

• Keep labels to stable, low-cardinality dimensions; never label by pod hash, commit, request ID, or timestamp.
• Use metric_relabel_configs to drop or normalize volatile labels at the vmagent/collector before ingestion.
• Map Kubernetes pod names to their controller (Deployment/StatefulSet) with relabeling so restarts do not mint new series.
• Track rate(vm_new_timeseries_created_total[5m]) on a dashboard and alert when it deviates from baseline.
• Review the cardinality explorer after each major instrumentation change.
• Prefer a fixed metric name with bounded labels over dynamically generated metric names.

Quick Command Reference

# Churn rate (run in vmui / MetricsQL)
# sum(rate(vm_new_timeseries_created_total[5m]))

# TSDB cardinality breakdown to find the churning label
curl -s 'http://localhost:8428/api/v1/status/tsdb'

# vmagent relabel snippet to drop a volatile label
# metric_relabel_configs:
# - action: labeldrop
#   regex: git_commit|request_id|session_id

# Watch new series creation counter
curl -s http://localhost:8428/metrics | grep vm_new_timeseries_created_total

Conclusion

“too high churn rate” is an early warning that your label design is minting throwaway series. Left alone it degrades into high memory use, slow queries, and eventually the maxUniqueTimeseries limit. Quantify it with vm_new_timeseries_created_total, pinpoint the volatile label in the cardinality explorer, and eliminate it with relabeling at ingestion. Fixing churn at the source keeps indexdb small and every downstream query fast.