Skip to content
CloudOps
Newsletter
All prompts
AI for Prometheus & Monitoring Difficulty: Advanced ClaudeChatGPT

Prometheus honor_labels & honor_timestamps Conflict Resolution Prompt

Diagnose and fix label collisions and timestamp drift caused by honor_labels/honor_timestamps when scraping federation endpoints, Pushgateway, or exporters that expose their own job/instance labels.

Target user
Platform engineers debugging duplicated, overwritten, or back-dated series in a Prometheus scrape pipeline
Difficulty
Advanced
Tools
Claude, ChatGPT

The prompt

You are a senior observability engineer who has untangled dozens of broken scrape pipelines where exporter-supplied labels silently clobbered Prometheus' own job and instance labels.

I will provide:
- The relevant scrape_config block (job_name, honor_labels, honor_timestamps, metrics_path)
- A raw sample of the exposition the target returns (with its label sets and any inline timestamps)
- The symptom (duplicate series, missing instance label, out-of-order samples, back-dated points, or wrong job attribution)

Your job:

1. **Explain the two flags precisely** — how `honor_labels: true` makes target-exposed `job`/`instance`/other reserved labels win over the scrape config, versus the default behavior of prefixing collisions with `exported_`; and how `honor_timestamps: true` accepts timestamps embedded in the exposition rather than stamping at scrape time.

2. **Map the conflict** — show exactly which labels in the provided exposition collide with the configured target labels and what the resulting series identity becomes under each flag setting.

3. **Diagnose the symptom** — tie the reported problem to a root cause: federation/Pushgateway needing `honor_labels: true`, an exporter wrongly emitting `instance`, or honored timestamps producing out-of-order or staleness-breaking samples.

4. **Prescribe the fix** — recommend the correct flag values for this target type and add `relabel_configs`/`metric_relabel_configs` to rename or drop offending labels (e.g. relabel `exported_job` back to `job`) where flags alone are insufficient.

5. **Timestamp safety** — explain when `honor_timestamps: true` is required (federation, cAdvisor) versus dangerous (clock-skewed exporters causing out-of-order rejections and broken staleness markers).

6. **Validation plan** — give `promtool` / `curl` steps and a PromQL check (`count by (__name__)` and label inspection) to confirm series identity and timestamp ordering after the change.

Output as: (a) a labeled before/after table of the affected series, (b) the corrected scrape_config YAML with inline comments, (c) any required relabel_configs, (d) the single most likely root cause stated in one sentence.

Do not enable honor_timestamps globally to "fix" one federation job — scope it to the specific target.
Newsletter

Free: the DevOps AI Incident-Triage Cheat Sheet

Subscribe and we’ll send you the one-page cheat sheet — plus weekly AI prompts, automation ideas, and tool reviews for infrastructure engineers. One email a week. No spam, unsubscribe anytime.

  • AI Incident-Triage Cheat Sheet (PDF)
  • Access to 1,603 DevOps AI prompts
  • One practical workflow email per week