🚨 OpenTelemetry Error Guide: 'duplicate metric with different label sets' — Fix Prometheus Receiver Conflicts

Overview

The OpenTelemetry prometheus receiver enforces that within a scrape, each metric name maps to a single consistent identity. When two samples share a name but disagree on label sets or type, the receiver rejects them and logs:

error: failed to scrape Prometheus endpoint
duplicate metric: "http_requests_total" with different label sets
scrape_pool: my-service  target: http://10.0.3.4:8080/metrics

A related conflict appears when the same metric name is exposed with different types across targets:

error: metric type conflict: "queue_size" is COUNTER but was previously GAUGE

The offending samples are dropped, so dashboards and alerts built on those series go blank.

Symptoms

• A specific metric is missing or partial in the backend while others scrape fine.
• Collector logs show duplicate metric or metric type conflict tied to a scrape pool/target.
• Started after adding a new exposed metric, an exporter upgrade, or merging two /metrics endpoints.
• otelcol_receiver_refused_metric_points increases for the prometheus receiver.
• Two libraries or subsystems in the same process export a same-named metric.

Common Root Causes

• Same name, different labels — two code paths register the same metric name with different label keys.
• Type conflict — the metric is a Counter in one place and a Gauge in another.
• Merged endpoints — a proxy or aggregator concatenates multiple /metrics outputs that collide.
• honor_labels interaction — target/instance labels overwrite or duplicate existing labels, producing indistinct series.
• Duplicate scrape jobs — two scrape_config entries hit the same target and merge samples.
• Library double-registration — a metric registered twice in one registry.

Diagnostic Workflow

First identify the exact metric and target from the log line, then scrape the raw endpoint to see the collision directly:

kubectl exec deploy/my-service -- wget -qO- http://localhost:8080/metrics | grep 'http_requests_total'
# Look for two lines with the same name but different label keys, or a # TYPE mismatch

Review the receiver config; a duplicate or overlapping scrape job is a common cause:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: my-service
          honor_labels: true      # avoid clobbering exposed labels with target labels
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_label_app]
              regex: my-service
              action: keep

Drop or rename one side of the collision with metric_relabel_configs when you can’t fix the source:

          metric_relabel_configs:
            # Drop the conflicting legacy series
            - source_labels: [__name__, subsystem]
              regex: "queue_size;legacy"
              action: drop

Confirm the receiver stopped refusing points after the change:

curl -s http://localhost:8888/metrics | grep -E 'receiver_refused_metric_points|receiver_accepted_metric_points'
journalctl -u otelcol -f | grep -i 'duplicate metric\|type conflict'

Example Root Cause Analysis

A service embedded two HTTP frameworks during a migration, and both registered http_requests_total — one with labels {method, path} and the other with {verb, route}. The raw /metrics output showed two blocks under the same name, and the prometheus receiver logged duplicate metric: "http_requests_total" with different label sets, dropping both and blanking the request-rate dashboard.

The real fix was in code: the legacy framework’s metric was renamed to http_requests_legacy_total, eliminating the collision. As an interim mitigation before the redeploy, a metric_relabel_configs drop on the legacy series with the verb label restored the primary dashboard immediately. After the redeploy, otelcol_receiver_refused_metric_points for the pool returned to zero.

Prevention Best Practices

• Enforce unique metric names per registry in code review; never let two subsystems export the same name with different labels or types.
• Keep label sets stable for a given metric name across all targets and versions; changing a metric’s type or labels is a breaking change.
• Use honor_labels deliberately and understand whether target labels should override exposed labels for your setup.
• Avoid overlapping scrape jobs; ensure each target is scraped by exactly one job.
• Alert on otelcol_receiver_refused_metric_points so silent metric drops from collisions surface fast.

Quick Command Reference

# Inspect the raw endpoint for colliding names/types
kubectl exec deploy/my-service -- wget -qO- localhost:8080/metrics | grep 'http_requests_total'

# Watch for the error live
journalctl -u otelcol -f | grep -i 'duplicate metric\|type conflict'

# Check refused vs accepted metric points
curl -s http://localhost:8888/metrics | grep -E 'receiver_refused_metric_points|receiver_accepted_metric_points'

# Validate config after relabel edits
otelcol validate --config /etc/otelcol/config.yaml

Conclusion

The duplicate metric with different label sets error means the prometheus receiver saw one metric name claiming two identities within a scrape. The durable fix is at the source — unique names and consistent types/labels — with metric_relabel_configs drops or renames as an interim mitigation. Watching receiver_refused_metric_points ensures future collisions are caught before they blank a dashboard.