Grafana Histogram & Heatmap Panel Design Prompt

You are a senior observability engineer who visualizes distributions (latency, size, duration) correctly rather than hiding them behind averages.

I will provide:
- The metric (classic `_bucket` histogram, Prometheus native histogram, or raw samples)
- The datasource and the question the panel must answer
- The expected value range and unit

Your job:

1. **Pick the right panel**: heatmap for distribution-over-time, histogram panel for a single-window distribution — and say why an average/percentile line alone is misleading here.
2. **Handle the bucket source correctly**:
   - Classic histograms: query `sum(rate(metric_bucket[$__rate_interval])) by (le)`, set the heatmap "Calculate from data / bucket bound = Upper (le)" and format as Heatmap.
   - Native histograms: query the raw metric and let the heatmap read native buckets — no `by (le)`.
3. **Bucketing**: warn when `le` buckets are too coarse to answer the question and recommend re-instrumenting the histogram boundaries.
4. **Color scaling**: choose linear vs exponential (log) color scale so a long tail is visible, and set the scheme and a sensible max so hot cells do not saturate.
5. **Exemplars**: enable exemplar display so a hot cell links to a trace where the datasource supports it.
6. **Units and axes**: set the Y unit (seconds/bytes) and bucket axis decimals so labels read cleanly.
7. **Provisioning**: deliver the panel JSON with the query, `le` handling, color scale, and unit encoded.

Mark DESTRUCTIVE: none — read-only. But warn that a heatmap over high-cardinality `le` series or a huge time range can be an expensive query.

---

Metric and type: [DESCRIBE]
Datasource and question: [DESCRIBE]
Value range and unit: [DESCRIBE]

Why this prompt works

Latency is a distribution, and averages/p99 lines quietly erase the multi-modal behavior that a heatmap makes obvious. The two failure modes are always the same: wrong le handling (classic vs native histogram) and a linear color scale that buries the tail. This prompt pins both down and delivers the panel as JSON so the bucketing and color scale are reproducible.

How to use it

1. Say whether the metric is a classic _bucket, native histogram, or raw samples — the query differs completely.
2. State the question (Is the tail growing? Is it bimodal?) so panel choice and color scale fit.
3. Give the unit and range so axes and decimals read cleanly.
4. Ask for the panel JSON with le handling and color scale encoded.

Useful commands

# Confirm the histogram buckets actually exist and their le boundaries
curl -s 'http://localhost:9090/api/v1/query?query=http_request_duration_seconds_bucket' \
  | jq -r '.data.result[].metric.le' | sort -g | uniq

# Import the panel-carrying dashboard
curl -X POST http://localhost:3000/api/dashboards/db \
  -H "Authorization: Bearer $GRAFANA_TOKEN" \
  -H "Content-Type: application/json" \
  -d @latency-heatmap.json

Example config

// Heatmap over a classic Prometheus histogram: bucket bound = Upper (le), log color scale
{
  "type": "heatmap",
  "title": "Request latency distribution",
  "targets": [
    {
      "expr": "sum(rate(http_request_duration_seconds_bucket[$__rate_interval])) by (le)",
      "format": "heatmap",
      "legendFormat": "{{le}}"
    }
  ],
  "options": {
    "calculate": false,
    "yAxis": { "unit": "s", "decimals": 2 },
    "color": { "mode": "scheme", "scheme": "Spectral", "scale": "exponential" },
    "cellGap": 1,
    "exemplars": { "color": "rgba(255,0,255,0.7)" }
  }
}

Common findings this catches

• Hidden tail → average/p99 line instead of a distribution.
• Double-counted buckets → by (le) used on a native histogram.
• Flat blob → linear color scale over long-tailed data.
• Unanswerable question → le boundaries too coarse to instrument.
• 1000x errors → unit mismatch between data and axis.

When to escalate

• Histogram re-instrumentation — the owning service team.
• Native-histogram enablement in Prometheus — the metrics platform team.
• Query cost on high-cardinality le series — the observability platform owner.