Skip to content
🎉 Launch sale:50% off everything over $22 — automatically applied at checkout· ends Aug 2Shop the sale →
DevOps AI ToolKit
Newsletter
All prompts
AI for Prometheus & Monitoring Difficulty: Advanced ClaudeChatGPT

Prometheus honor_labels & honor_timestamps Conflict Resolution Prompt

Diagnose and fix label collisions and timestamp drift caused by honor_labels/honor_timestamps when scraping federation endpoints, Pushgateway, or exporters that expose their own job/instance labels.

Target user
Platform engineers debugging duplicated, overwritten, or back-dated series in a Prometheus scrape pipeline
Difficulty
Advanced
Tools
Claude, ChatGPT

The prompt

You are a senior observability engineer who has untangled dozens of broken scrape pipelines where exporter-supplied labels silently clobbered Prometheus' own job and instance labels.

I will provide:
- The relevant scrape_config block (job_name, honor_labels, honor_timestamps, metrics_path)
- A raw sample of the exposition the target returns (with its label sets and any inline timestamps)
- The symptom (duplicate series, missing instance label, out-of-order samples, back-dated points, or wrong job attribution)

Your job:

1. **Explain the two flags precisely** — how `honor_labels: true` makes target-exposed `job`/`instance`/other reserved labels win over the scrape config, versus the default behavior of prefixing collisions with `exported_`; and how `honor_timestamps: true` accepts timestamps embedded in the exposition rather than stamping at scrape time.

2. **Map the conflict** — show exactly which labels in the provided exposition collide with the configured target labels and what the resulting series identity becomes under each flag setting.

3. **Diagnose the symptom** — tie the reported problem to a root cause: federation/Pushgateway needing `honor_labels: true`, an exporter wrongly emitting `instance`, or honored timestamps producing out-of-order or staleness-breaking samples.

4. **Prescribe the fix** — recommend the correct flag values for this target type and add `relabel_configs`/`metric_relabel_configs` to rename or drop offending labels (e.g. relabel `exported_job` back to `job`) where flags alone are insufficient.

5. **Timestamp safety** — explain when `honor_timestamps: true` is required (federation, cAdvisor) versus dangerous (clock-skewed exporters causing out-of-order rejections and broken staleness markers).

6. **Validation plan** — give `promtool` / `curl` steps and a PromQL check (`count by (__name__)` and label inspection) to confirm series identity and timestamp ordering after the change.

Output as: (a) a labeled before/after table of the affected series, (b) the corrected scrape_config YAML with inline comments, (c) any required relabel_configs, (d) the single most likely root cause stated in one sentence.

Do not enable honor_timestamps globally to "fix" one federation job — scope it to the specific target.

Run this prompt with AI

Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.

Related prompts

More Prometheus & Monitoring prompts & error guides

Browse every Prometheus & Monitoring prompt and troubleshooting guide in one place.

Free download · 368-page PDF

Reading prompts? Get all 500 in one free PDF

500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.

  • 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
  • Instant PDF download — yours free, forever
  • Plus one practical AI-workflow email a week (no spam)

Single opt-in · unsubscribe anytime · no spam.