🚨 Prometheus Error Guide: 'duplicate sample for timestamp' Colliding Label Sets

Exact Error Message

duplicate sample for timestamp is a TSDB append-path rejection logged during a scrape or remote-write:

ts=2026-06-27T10:41:18.227Z caller=scrape.go:1654 level=warn component="scrape manager" scrape_pool=node msg="Error on ingesting samples with different value but same timestamp" num_dropped=7

The TSDB-level error string is:

err="duplicate sample for timestamp"

err="duplicate sample for timestamp" series="node_filesystem_size_bytes{instance=\"10.0.4.2:9100\",job=\"node\"}"

The scrape returns HTTP 200, but the colliding samples are dropped — the affected series gets gaps even though the target shows up.

What the Error Means

Within a single scrape (or a single remote-write request), Prometheus expects every series — a unique combination of __name__ plus labels — to appear at most once. If the same series appears twice in one payload with the same timestamp but a different value, Prometheus cannot decide which value is correct and rejects the duplicate with duplicate sample for timestamp.

This is fundamentally a label uniqueness problem, not a timing problem. It differs from out of order sample (older timestamp than the last sample of a series) and from out of bounds (timestamp outside the head’s valid window). Here the timestamps are identical and valid — the issue is two data points claiming to be the same series at the same instant.

Common Causes

• An exporter exposing the same series twice. A bug or misconfiguration prints two lines with identical __name__ and labels in one /metrics response.
• Two targets producing identical label sets. Two endpoints relabeled to the same job/instance, or honor_labels letting a federated/pushed instance override and collide.
• Pushgateway duplicates. Two jobs pushing the same grouping key, or a push that re-sends a metric already present under the same labels.
• Relabeling that collapses labels. A labeldrop/labelmap rule removing a distinguishing label (instance, pod, replica) so two distinct series merge into one.
• Federation overlap. A /federate scrape pulling series that the local Prometheus also produces, with honor_labels: true causing collisions.
• Missing instance. A static or file-SD target list with multiple entries that all resolve to the same instance label.

How to Reproduce the Error

Stand up a tiny exporter that prints the same series twice, then scrape it:

cat > /tmp/dup_metrics.txt <<'EOF'
# HELP demo_value A demo gauge
# TYPE demo_value gauge
demo_value{env="prod"} 1
demo_value{env="prod"} 2
EOF
python3 -m http.server 9999 --directory /tmp &   # serves /dup_metrics.txt

Point a scrape job at it and watch the log:

journalctl -u prometheus -f | grep -i 'duplicate sample'

level=warn msg="Error on ingesting samples with different value but same timestamp" num_dropped=1

The two demo_value{env="prod"} lines are the same series with different values — exactly the collision.

Diagnostic Commands

Grep a suspect target’s raw output for duplicate metric lines (same name+labels appearing more than once):

curl -s http://10.0.4.2:9100/metrics \
  | grep -v '^#' \
  | sed -E 's/ [0-9.eE+-]+$//' \
  | sort | uniq -d

node_filesystem_size_bytes{device="/dev/sda1",mountpoint="/",fstype="ext4"}

Any line printed by uniq -d is exposed twice and will collide.

Find series Prometheus already sees more than once per evaluation:

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=count by (__name__, job, instance) ({__name__=~".+"}) > 1' \
  | jq -r '.data.result[] | "\(.metric.__name__)\t\(.metric.job)\t\(.value[1])"' | head

Look for duplicate active targets resolving to the same labels:

curl -s http://localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | [.labels.job,.labels.instance,.scrapeUrl] | @tsv' \
  | sort | uniq -d

Check whether honor_labels or label-dropping relabel rules are in play:

grep -nE 'honor_labels|labeldrop|labelmap|target_label' /etc/prometheus/prometheus.yml

Watch the duplicate-timestamp rejection counter:

rate(prometheus_target_scrapes_sample_duplicate_timestamp_total[5m]) > 0

Step-by-Step Resolution

1. Dedupe the exporter. If uniq -d showed repeated lines, fix the exporter so each series appears once. For a custom exporter this usually means a registry/collector registering the same metric twice, or a loop emitting a label that is not actually distinct.

2. Restore a distinguishing label. If two replicas collapsed into one series, stop dropping the label that separates them:

# WRONG - merges all replicas into one series
metric_relabel_configs:
  - action: labeldrop
    regex: instance

Remove that rule (or replace instance with a non-distinguishing label) so each replica keeps a unique instance.

3. Ensure unique instance/job per target. For static/file SD, give each entry a distinct instance via relabeling:

relabel_configs:
  - source_labels: [__address__]
    target_label: instance

4. Use honor_labels correctly. For Pushgateway and federation, honor_labels: true keeps the source’s labels; only enable it where the source’s instance/job are authoritative and unique, otherwise leave it false so Prometheus assigns distinct ones:

- job_name: pushgateway
  honor_labels: true
  static_configs:
    - targets: ['pushgateway:9091']

5. Fix Pushgateway grouping. Push each job under a unique grouping key (/metrics/job/<job>/instance/<instance>) so two jobs cannot write the same series.

6. Trim federation overlap. Make /federate match[] selectors exclude series the local server also produces, and avoid scraping a hierarchical Prometheus for metrics it in turn scraped from the same exporters you scrape directly.

7. Verify the fix. After changing the exporter, relabeling, or honor_labels, re-run the duplicate check and confirm the counter stops climbing:

curl -s http://10.0.4.2:9100/metrics | grep -v '^#' | sed -E 's/ [0-9.eE+-]+$//' | sort | uniq -d
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=increase(prometheus_target_scrapes_sample_duplicate_timestamp_total[5m])' \
  | jq -r '.data.result[] | .value[1]'

A clean (empty) uniq -d and a flatlined counter confirm the collision is gone.

Prevention and Best Practices

• Each series must come from exactly one source. Audit for the same endpoint scraped under two jobs, or two Services pointing at one pod.
• Never labeldrop an identity label (instance, pod, replica) on a multi-target job.
• Add a CI check that runs curl /metrics | grep -v '^#' | sed 's/ [0-9.eE+-]*$//' | sort | uniq -d against new exporters and fails on any output.
• Use honor_labels: true only for Pushgateway/federation, and ensure those sources carry unique instance/job.
• Alert on prometheus_target_scrapes_sample_duplicate_timestamp_total so silent drops surface immediately.

Related Errors

• out of order sample — same series, older timestamp than the last stored sample. A timing problem, not a uniqueness one.
• out of bounds — timestamp outside the head’s valid window (too old / too far future). Clock and backfill territory.
• Error on ingesting samples with different value but same timestamp — the human-readable log wrapper around this exact error.

Frequently Asked Questions

What’s the difference between duplicate and out-of-order samples? duplicate sample for timestamp means two samples for the same series share the same timestamp but have different values (a label-uniqueness problem). out of order sample means a sample arrived with a timestamp older than the previous sample of that series (a timing problem). Different causes, different fixes.

Why does up show healthy if samples are being dropped? up reflects whether the scrape’s HTTP request succeeded, which it did. The duplicate rejection happens later, at the TSDB append step, so the target looks healthy while specific series get gaps.

Could honor_labels be causing this? Yes. With honor_labels: true, the source’s instance/job are preserved. If two pushed/federated sources carry the same instance/job, their series collide. Disable honor_labels (so Prometheus assigns unique labels) unless the sources are guaranteed unique.

How do I quickly confirm the offending exporter? Run curl -s <target>/metrics | grep -v '^#' | sed -E 's/ [0-9.eE+-]+$//' | sort | uniq -d. Any line it prints is exposed more than once and is the duplicate.