🚨 Prometheus Error Guide: 'vector cannot contain metrics with the same labelset' — Fix Duplicate Series

Overview

PromQL raises this error at evaluation time when the result of an expression contains two or more series that share an identical label set. An instant vector must have unique label sets, so the engine refuses the result instead of guessing which one you meant:

execution: vector cannot contain metrics with the same labelset

It surfaces in the Prometheus UI, in Grafana panels as a query error, and in the rule evaluation log when a recording or alerting rule produces the collision:

level=warn component=rule_group rule="job:http_requests:rate5m"
  msg="Evaluating rule failed" err="vector cannot contain metrics with the same labelset"

Symptoms

• A query works on some time ranges and fails on others (the collision only exists when both series have data).
• A label_replace, label_join, aggregation without by, or a binary operation produces the error.
• A recording rule’s series is missing and its rule group shows evaluation failures.
• The base metric queries fine; the error only appears after a transform or join is added.
• Grafana shows “vector cannot contain metrics with the same labelset” instead of data.

Common Root Causes

• Dropping a distinguishing label — sum without (instance) or sum by (job) collapses several series that differ only by the removed label into one label set.
• label_replace overwriting a label — rewriting instance (or any label) to a constant/derived value makes previously-distinct series identical.
• label_join producing collisions — joining labels into a new one that is not actually unique.
• Aggregating away a label that a later on()/group_left still needs, so two rows land on the same key.
• Recording rule output labels — the rule’s implicit output labels plus the query result collide (e.g. two source series map to the same recorded series).
• Mixing two metrics with or where both contribute a series with the same labels.

Diagnostic Workflow

Start by running just the inner expression that feeds the failing operation and count how many series collapse onto each label set. This count by trick finds the collision directly — any group with a value greater than 1 is a colliding label set:

count by (job, code) (
  sum without (instance) (rate(http_requests_total[5m]))
) > 1

Inspect the raw series before the transform to see which label actually distinguishes them:

rate(http_requests_total{job="api"}[5m])

If a label_replace is involved, evaluate it and check whether the target label is now constant across formerly-distinct series:

label_replace(node_cpu_seconds_total, "instance", "node", "instance", ".*")

Confirm which rule is failing and read the evaluator error from the server:

# List rule evaluation failures
curl -sS http://localhost:9090/api/v1/rules | \
  grep -o '"health":"[^"]*"' | sort | uniq -c

# Tail the evaluation error
journalctl -u prometheus --since '10 min ago' | grep -i 'same labelset'

Validate the corrected rule file before reloading:

promtool check rules /etc/prometheus/rules/http.yml

Example Root Cause Analysis

A recording rule aggregated request rates across instances:

sum without (instance, pod) (rate(http_requests_total[5m]))

The source metric carried both instance and pod, but two Deployments in different namespaces exposed the same job, handler, and code labels. After stripping instance and pod, the only remaining labels were job, handler, and code — and both namespaces’ series now had identical label sets. On time ranges where only one namespace had traffic the rule passed; once both were active, evaluation failed with vector cannot contain metrics with the same labelset.

Running count by (job, handler, code) (...) > 1 returned a value of 2, confirming the collision. The fix was to keep the distinguishing label instead of dropping it:

sum by (namespace, job, handler, code) (rate(http_requests_total[5m]))

Switching from without (instance, pod) to an explicit by (...) that retains namespace gave every result row a unique label set, and the rule group went healthy on the next evaluation.

Prevention Best Practices

• Prefer explicit by (...) over without (...) in aggregations, so you consciously choose the output labels and cannot accidentally collapse a distinguishing one.
• Before label_replace/label_join, confirm the target label is genuinely unique across the input series with a count by (...) > 1 check.
• Keep a namespace/cluster/env label in aggregations that span multiple environments to prevent cross-environment collisions.
• Unit-test recording rules with promtool test rules using input series that include the multi-source case, not just a single instance.
• Run promtool check rules in CI so a query that can collide is caught before it reaches the evaluator.
• When joining with on()/group_left, verify the match key is unique on the many-side to avoid producing duplicate rows.

Quick Command Reference

# Find colliding label sets (any group > 1 is a collision)
count by (LABELS_YOU_KEEP) ( YOUR_INNER_EXPRESSION ) > 1

# Inspect raw series to see the distinguishing label
rate(http_requests_total{job="api"}[5m])

# Which rules are failing?
curl -sS http://localhost:9090/api/v1/rules | grep -o '"health":"[^"]*"' | sort | uniq -c

# Validate + unit-test rules before reload
promtool check rules /etc/prometheus/rules/http.yml
promtool test rules /etc/prometheus/tests/http_test.yml

Conclusion

vector cannot contain metrics with the same labelset is PromQL enforcing that instant vectors have unique label sets. The culprit is almost always an aggregation or label rewrite that erased the one label distinguishing two series. Use count by (...) > 1 to pinpoint the colliding key, then rewrite the query to retain that label — favoring explicit by (...) — and guard rule files with promtool check and test so the collision cannot reappear the next time a second source starts reporting data.