💥 Taming Prometheus Metric Cardinality Before It Tames You

Almost every Prometheus server I’ve seen fall over died the same way: cardinality. Someone added a label with a user ID, or a request path that includes a UUID, and the series count exploded from thousands to millions. Memory spikes, queries crawl, the server OOMs. Cardinality is the silent killer of Prometheus, and once you understand it, it’s entirely preventable.

What cardinality actually is

Prometheus stores one time series for every unique combination of metric name and label values. A metric with two labels, each with ten possible values, is 100 series. Sounds harmless.

Now make one label a user ID with a million values:

http_requests_total{method="GET", user_id="...one of a million..."}

That’s a million series for one metric. Each series costs memory — roughly a few KB resident, plus index overhead. Multiply by your other metrics and you’re out of RAM. The rule: a label’s value set must be bounded and small. Bounded means “you can name all the possible values.” Unbounded labels are the disease.

The labels that will hurt you

These are the repeat offenders. Never put them in a label:

• User IDs, session IDs, request IDs, trace IDs — unbounded by definition.
• Full URL paths with embedded IDs — /users/8a3f.../orders/91c2.... Normalize to /users/:id/orders/:id.
• Email addresses, IP addresses — unbounded.
• Error messages as labels — they vary infinitely. Use an error code instead.
• Timestamps or anything monotonic — every value is unique.

If a label’s value comes from user input or grows with traffic, it does not belong in a metric. That data belongs in logs or traces, where high cardinality is the whole point. Metrics are for aggregates.

Find your cardinality offenders

Prometheus exposes its own internals. Start here:

# how many series each metric has (top offenders first)
topk(10, count by (__name__)({__name__=~".+"}))

# total series in the head block
prometheus_tsdb_head_series

The TSDB status page (/tsdb-status in newer versions, or the runtime API) gives you the highest-cardinality metric names and label values directly. When a server is struggling, that page usually names the culprit in thirty seconds.

For a specific metric, find which label is blowing it up:

# distinct values of a label for one metric
count(count by (path) (http_requests_total))

If that returns 400,000, your path label is unbounded. Found it.

Drop or relabel at ingestion

The cleanest fix is to never ingest the bad label. Use metric_relabel_configs in your scrape config to drop labels or whole metrics before they hit storage:

metric_relabel_configs:
  # drop a high-cardinality label entirely
  - regex: 'user_id'
    action: labeldrop
  # normalize paths: collapse numeric IDs to a placeholder
  - source_labels: [path]
    regex: '/users/[0-9a-f]+'
    target_label: path
    replacement: '/users/:id'
  # drop an entire noisy metric you don't use
  - source_labels: [__name__]
    regex: 'go_gc_duration_seconds.*'
    action: drop

labeldrop removes a label (and merges the now-identical series). Path normalization is the highest-value relabel I know — it turns millions of unique paths into a handful of route templates. Do this at scrape time and the bad data never costs you a byte of storage.

Fix it at the source when you can

Relabeling is a bandage. The real fix is the instrumentation. If you control the code, don’t emit the bad label at all. Use route templates (/users/:id) not raw paths. Use error codes not messages. Bound your label values to a known set defined in code.

A useful discipline: when you add a label, ask “how many distinct values can this ever have?” If you can’t answer with a small number, don’t add it. This one question prevents most cardinality incidents before they happen.

Watch the trend, not just the moment

Cardinality creep is gradual — a label added here, a new dimension there — until one day you OOM. Alert on the trajectory:

- alert: HeadSeriesGrowing
  expr: prometheus_tsdb_head_series > 2000000
  for: 30m
  labels:
    severity: ticket
  annotations:
    summary: "Prometheus head series above 2M, check for cardinality creep"

Catching it at “growing” instead of “exploded” means a calm investigation instead of a 3am OOM loop.

The aggregation-vs-detail tradeoff

Sometimes you genuinely want per-customer breakdowns. The answer isn’t to jam customer IDs into your main metrics — it’s to keep a separate, deliberately small metric for the handful of top customers, or to push that detail into a system built for high cardinality (logs, traces, or a columnar store). Keep your hot Prometheus metrics aggregate and bounded; put the needles in the haystack somewhere designed for needles.

Where AI helps

Spotting a problematic label in a scrape config or instrumentation diff is pattern-matching, and AI is quick at it. I’ll paste a scrape config or a chunk of instrumentation and ask it to flag any label that looks unbounded — IDs, paths, messages — and propose the metric_relabel_configs to neutralize them. It also drafts the path-normalization regexes, which are tedious to get right by hand.

You still verify against your real series counts, but it’s a fast second pair of eyes. We keep monitoring prompts for cardinality audits, and the rules our Alert Rule Generator produces avoid unbounded labels by construction.

The bottom line

Cardinality is the one Prometheus failure mode that bites everyone eventually. The defense is simple and boring: bounded labels only, normalize paths, drop unbounded dimensions at ingestion, and watch your head-series count trend upward before it cliffs. Do that, and your Prometheus stays fast, cheap, and alive — which is more than a lot of clusters can say.

Generated relabel configs and audits are assistive, not authoritative. Always verify against your real series counts and test scrape config changes in staging first.