PromQL Native Histogram histogram_count & histogram_sum Debugging Prompt
Debug quantiles, averages, and rates over Prometheus native histograms using histogram_count, histogram_sum, histogram_fraction, and histogram_quantile correctly.
- Target user
- Engineers querying native histograms whose quantiles or averages look wrong
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a Prometheus query expert who reasons precisely about native histogram samples, which are a single complex sample per series rather than a fan-out of _bucket series. Treat classic-histogram intuition as a trap. I will provide: - The native histogram metric name and a sample of its label set: [METRIC + LABELS] - The query I wrote and the result that looks wrong: [QUERY + UNEXPECTED OUTPUT] - What I expected (a quantile, an average, a fraction below a threshold, a rate): [EXPECTATION] - My Prometheus version and whether native histograms are enabled via --enable-feature=native-histograms: [VERSION + FLAGS] Your job: 1. **Confirm it is actually a native histogram** — explain how to tell (no _bucket/_count/_sum suffix series; the raw sample is a histogram). Show the query to inspect it and warn that mixing native and classic exposition for the same name corrupts results. 2. **Map the function to the math** — for my goal, pick the correct function: - average = histogram_sum(rate(metric[5m])) / histogram_count(rate(metric[5m])) - quantile = histogram_quantile(0.95, rate(metric[5m])) - fraction below a value = histogram_fraction(0, [VALUE], rate(metric[5m])) Explain why rate() must wrap the histogram before these functions for counter-style histograms. 3. **Diagnose the wrong result** — walk the most common causes: forgetting rate() (giving cumulative since process start), applying classic-histogram sum(rate(..._bucket)) syntax to a native histogram, or schema/resolution mismatch after a scrape change. 4. **Show the corrected query** with inline annotation of what each function consumes and emits. 5. **Sanity-check bounds** — give me a cheap cross-check (for example, the p99 must lie between min and max observed, and histogram_count(rate) must roughly equal rate of request count). Output as: (a) classification of the metric, (b) the corrected query, (c) a 4-row table mapping goal to function, (d) one cross-check query I can run to prove the answer is sane. Show the reasoning about which function consumes a histogram vs a float. Never silently switch between native and classic syntax.
Why this prompt works
Native histograms changed the mental model for histogram queries, and most engineers carry classic-histogram habits straight into the new world. With classic histograms you summed _bucket series and fed them to histogram_quantile; with native histograms a single sample already carries the full distribution, so the function consumes the histogram directly. The prompt forces the model to first classify the metric — because the single most common failure is applying classic syntax to native data (or vice versa) and getting empty or nonsensical output. By making classification step one, you stop the model from confidently writing a query against series that do not exist.
The prompt also anchors on the rate() requirement, which is the subtlest trap. histogram_quantile, histogram_sum, and histogram_count will happily run over a raw counter histogram and return values computed from the cumulative distribution since the process started, which looks plausible but drifts as the process ages. Demanding that the model justify the rate() wrapper, and tie each function to its exact use case in a table, turns a guessy “try this query” answer into something you can reason about line by line.
Finally, the cross-check requirement matters because native histogram bugs are quiet. There is no error — just a quantile that is subtly off. Requiring a cheap independent verification query (p99 bounded by observed min/max, histogram_count tracking request rate) gives you a falsifiable test instead of trust. That is the AI-drafts, human-verifies loop applied to a place where the failure mode is silence, not a stack trace.