Prometheus honor_labels & honor_timestamps Conflict Resolution Prompt
Diagnose and fix label collisions and timestamp drift caused by honor_labels/honor_timestamps when scraping federation endpoints, Pushgateway, or exporters that expose their own job/instance labels.
- Target user
- Platform engineers debugging duplicated, overwritten, or back-dated series in a Prometheus scrape pipeline
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT
The prompt
You are a senior observability engineer who has untangled dozens of broken scrape pipelines where exporter-supplied labels silently clobbered Prometheus' own job and instance labels. I will provide: - The relevant scrape_config block (job_name, honor_labels, honor_timestamps, metrics_path) - A raw sample of the exposition the target returns (with its label sets and any inline timestamps) - The symptom (duplicate series, missing instance label, out-of-order samples, back-dated points, or wrong job attribution) Your job: 1. **Explain the two flags precisely** — how `honor_labels: true` makes target-exposed `job`/`instance`/other reserved labels win over the scrape config, versus the default behavior of prefixing collisions with `exported_`; and how `honor_timestamps: true` accepts timestamps embedded in the exposition rather than stamping at scrape time. 2. **Map the conflict** — show exactly which labels in the provided exposition collide with the configured target labels and what the resulting series identity becomes under each flag setting. 3. **Diagnose the symptom** — tie the reported problem to a root cause: federation/Pushgateway needing `honor_labels: true`, an exporter wrongly emitting `instance`, or honored timestamps producing out-of-order or staleness-breaking samples. 4. **Prescribe the fix** — recommend the correct flag values for this target type and add `relabel_configs`/`metric_relabel_configs` to rename or drop offending labels (e.g. relabel `exported_job` back to `job`) where flags alone are insufficient. 5. **Timestamp safety** — explain when `honor_timestamps: true` is required (federation, cAdvisor) versus dangerous (clock-skewed exporters causing out-of-order rejections and broken staleness markers). 6. **Validation plan** — give `promtool` / `curl` steps and a PromQL check (`count by (__name__)` and label inspection) to confirm series identity and timestamp ordering after the change. Output as: (a) a labeled before/after table of the affected series, (b) the corrected scrape_config YAML with inline comments, (c) any required relabel_configs, (d) the single most likely root cause stated in one sentence. Do not enable honor_timestamps globally to "fix" one federation job — scope it to the specific target.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Prometheus keep_dropped_targets Limit Tuning Prompt
Set keep_dropped_targets on service-discovery-heavy Prometheus servers to cap the memory spent retaining metadata for relabel-dropped targets, so a churny SD source (Kubernetes, EC2, Consul) can't quietly balloon scrape-manager memory or the /api/v1/targets response.
-
Prometheus http_sd Dynamic Target Discovery Prompt
Design and debug an http_sd_config integration so Prometheus pulls its scrape targets from a custom HTTP discovery endpoint, with correct refresh, labeling, and failure handling.
-
Prometheus OAuth2 Scrape Target Authorization Prompt
Configure and troubleshoot OAuth2 client-credentials authorization on Prometheus scrape jobs so that Prometheus can pull /metrics from targets sitting behind an OIDC-protected gateway without leaking secrets or hammering the token endpoint.
-
Prometheus Scrape Config Relabel Target Pruning Design Prompt
Design relabel_configs in a scrape job to keep/drop the right targets from service discovery, rewrite the instance/job labels, and prune noisy discovered endpoints before they ever scrape.
More Prometheus & Monitoring prompts & error guides
Browse every Prometheus & Monitoring prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.