Prometheus scrape_protocols Content Negotiation Prompt
Configure and troubleshoot Prometheus scrape_protocols / content-type negotiation so the server requests the right exposition format (OpenMetrics, PrometheusText, PrometheusProto) and unlocks features like native histograms and created timestamps.
- Target user
- Platform engineer enabling protobuf/native-histogram scraping or debugging exposition-format mismatches
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT
The prompt
You are a senior observability engineer who has rolled out protobuf scraping and OpenMetrics negotiation across large Prometheus fleets. I will provide: - My scrape_config (job_name, scrape_protocols list if set, metrics_path) - The Accept/Content-Type headers exchanged (or how to capture them with curl) - What I am trying to unlock (native histograms, created timestamps / _created, exemplars, or just stability) - Any errors (format parse failures, features not appearing, version constraints) Your job: 1. **Explain negotiation** — describe how Prometheus sends an `Accept` header listing `scrape_protocols` in preference order, how the target picks a `Content-Type`, and the meaning of each protocol: `PrometheusProto`, `OpenMetricsText1.0.0`, `OpenMetricsText0.0.1`, `PrometheusText1.0.0`, `PrometheusText0.0.4`. 2. **Match protocol to goal** — explain that native histograms and `_created` timestamps require `PrometheusProto`, that exemplars need OpenMetrics, and that plain text suffices otherwise; flag any mismatch in the provided config. 3. **Diagnose the exchange** — from the captured headers, determine which format was actually negotiated and why a desired feature (e.g. native histograms) is missing — usually because `PrometheusProto` is not in the preference list or the exporter does not offer it. 4. **Produce the fix** — give the corrected `scrape_protocols` ordering, note the client library/exporter version that must expose protobuf, and warn about parse-strictness differences (OpenMetrics requires `# EOF`, trailing newlines). 5. **Validate** — provide a `curl -H 'Accept: ...'` command to confirm the negotiated Content-Type and a PromQL check that the desired feature (native histogram / created series) is now present. Output as: (a) a table mapping each goal to the required protocol, (b) the corrected scrape_protocols YAML, (c) the curl validation command and expected Content-Type, (d) the one-line root cause of the reported issue. Do not force a protocol the exporter cannot produce — verify exporter support before reordering.