Prometheus Histogram Bucket Boundary Design Prompt
Choose histogram bucket boundaries that match your SLO thresholds and latency distribution so quantile estimates are accurate where it matters, without exploding series cardinality from too many buckets.
- Target user
- Engineers instrumenting latency and size metrics
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT
The prompt
You are a senior observability engineer who knows that a histogram_quantile is only as honest as the bucket boundaries beneath it. I will provide: - The metric I am bucketing (request latency, payload size, queue depth) and its rough distribution - My SLO thresholds and the quantiles I report (p50/p95/p99) - My current `le` buckets and how many label combinations the metric has Your job: 1. **Anchor buckets to SLO thresholds** — ensure a bucket boundary sits exactly on each SLO target (e.g. 0.3s) so SLO compliance is read directly, not interpolated. 2. **Match buckets to the distribution** — explain why exponential/`ExponentialBuckets` fits long-tail latency and where linear buckets waste resolution. 3. **Estimate quantile error** — show how `histogram_quantile` linearly interpolates within a bucket and how wide buckets near the tail inflate p99 error. 4. **Budget cardinality** — compute series = buckets x label combinations and warn when bucket counts multiply against high-cardinality labels. 5. **Consider native histograms** — compare classic fixed buckets vs native (sparse) histograms for resolution-without-cardinality, and the migration tradeoff. 6. **Validate against real data** — propose a query to check how observations actually spread across current buckets and where boundaries are wasted or missing. Output as: (a) a recommended `le` bucket list with rationale per boundary, (b) the cardinality math for my labels, (c) the p95/p99 error implication, (d) a native-histogram recommendation if warranted. Warn clearly: quantiles read from coarse buckets can be confidently wrong — never report a p99 the bucket layout cannot actually resolve.