Grafana Dashboard Optimization with Recording Rules Prompt

You are a senior SRE who makes slow Grafana dashboards fast by pushing expensive PromQL into Prometheus recording rules and rewriting panels to read the pre-aggregated series.

I will provide:
- The slow panels' PromQL and the query inspector timings
- The scrape interval and Prometheus/Mimir scale
- Which labels the panels group by and which template variables drive them

Your job:

1. **Find the hot spots**: identify which panel expressions are expensive — high-cardinality `sum by`, long-range `rate()`, `histogram_quantile` over raw buckets, subqueries — and explain why each is slow (samples scanned, series touched).
2. **Design recording rules**: write Prometheus recording rules that pre-aggregate the expensive parts, following the `level:metric:operation` naming convention, at an evaluation interval matched to the scrape interval; keep recorded cardinality low by dropping labels the dashboard doesn't need.
3. **Rewrite panels**: show each panel's PromQL rewritten to read the recorded series, preserving the template-variable filters and `$__rate_interval` where still needed.
4. **Guard correctness**: ensure the recorded series still lets panels filter by the variables (don't aggregate away a label a variable needs); note where a rule must keep a label for drilldown.
5. **Validate**: give commands to `promtool check rules`, confirm the recorded series exist, and compare query inspector timings before/after.
6. **Deliver**: the recording rules YAML plus the updated panel queries, and note the trade-off (freshness = evaluation interval, extra TSDB series).

Mark DESTRUCTIVE: none — but warn that over-aggregating a recording rule permanently loses labels the dashboard can no longer filter on.

---

Slow panels' PromQL + timings: [DESCRIBE]
Scrape interval + scale: [DESCRIBE]
Group-by labels + variables: [DESCRIBE]

Why this prompt works

Slow dashboards are almost always a few panels re-running expensive aggregations on every refresh across every viewer. This prompt finds those hot spots from the query inspector, moves the heavy math into recording rules with correct naming and interval, rewrites the panels to read the recorded series, and validates the before/after so the speedup is proven rather than assumed.

How to use it

1. Bring the query inspector timings so the expensive panels are identified by data.
2. List the group-by labels and variables so no filter is aggregated away.
3. Give the scrape interval so the rule evaluation interval matches.
4. Compare timings after to confirm the load actually dropped.

Useful commands

# Validate recording rules before loading
promtool check rules recording-rules.yml

# Confirm the recorded series exist and are fresh
curl -s -G http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=job:http_request_errors:ratio_rate5m' | jq '.data.result | length'

Example config

Recording rule replacing a heavy per-panel error-ratio:

groups:
  - name: dashboard-accel
    interval: 15s
    rules:
      - record: job:http_request_errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))

Panel query, rewritten to read the rule:

job:http_request_errors:ratio_rate5m{job="$job"}

Common findings this catches

• Broken variable filter → label aggregated away in the rule.
• Gappy recorded series → evaluation interval ≠ scrape interval.
• Cost moved, not saved → over-broad by() in the rule.
• Wrong percentiles → histogram_quantile recorded at wrong level.
• Silent recording failure → rules shipped without promtool check.

When to escalate

• TSDB capacity for added series — the metrics platform team.
• Rule ownership and naming standards — the Prometheus rule owner.
• Freshness requirements vs interval trade-off — the service owner.