Grafana $__rate_interval Correctness Review Prompt
Review Grafana panel queries that use rate() to confirm they use $__rate_interval correctly, so dashboards stay accurate across zoom levels and scrape intervals.
- Target user
- Engineers building Grafana dashboards on Prometheus data
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a Grafana and PromQL expert who knows the difference between $__interval, $__range, and $__rate_interval, and why using the wrong one makes rate() panels lie when users zoom in or out. I will provide: - The panel queries that use rate/increase/irate: [QUERIES] - The scrape interval of the underlying job(s): [SCRAPE INTERVAL] - The dashboard's typical time ranges (live 15m, daily, 30-day): [RANGES] - Any complaint (graph goes flat/empty when zoomed in, spikes vanish when zoomed out): [SYMPTOM] Your job: 1. **Explain the three variables** — in one line each: - $__interval: the time per pixel/step Grafana computes from range and width. - $__range: the full selected range. - $__rate_interval: max($__interval + scrape_interval, 4 * scrape_interval), purpose-built so rate() always sees at least a few samples. State that rate() windows should almost always use $__rate_interval, not $__interval. 2. **Find the bug** — flag any rate()/increase() using a hardcoded window (rate(x[5m])) or $__interval. Explain the concrete failure: with $__interval, zooming in can shrink the window below the scrape interval so rate() sees <2 samples and returns empty; this is the "flat/empty when zoomed in" symptom. 3. **Rewrite correctly** — convert the queries to use $__rate_interval, and note where a fixed window is actually intentional (e.g. an alert recording rule that must match the alert's rate window). 4. **Handle the scrape-interval dependency** — confirm Grafana knows the scrape interval (via the data source "Scrape interval" setting or Min step) so $__rate_interval computes correctly; a wrong value silently breaks it. 5. **Verify** — give me the steps to prove the fix: load the panel at 15m and at 30d and confirm the shape is consistent and no gaps appear. Output as: (a) a 3-row table defining the variables, (b) a per-query verdict (ok / change to $__rate_interval / leave fixed and why), (c) the rewritten queries, (d) the data-source setting to check and the 2-step verification. Default rate windows to $__rate_interval unless there is a stated reason for a fixed window. Never recommend $__interval for a rate() window.
Why this prompt works
The $__rate_interval versus $__interval distinction is one of the highest-impact, least-understood details in Grafana-on-Prometheus dashboards. $__interval is the step Grafana derives from the time range and panel width, and it feels like the natural thing to put inside rate(). But when a user zooms in, $__interval shrinks, and once it drops below roughly twice the scrape interval, rate() no longer has two samples to work with and returns nothing. The panel goes flat or empty with no error message, the user assumes the service stopped, and trust in the dashboard evaporates. This prompt names the exact mechanism so the reviewer can connect the “empty when zoomed in” complaint to its real cause instead of blaming the data.
$__rate_interval exists specifically to solve this: it is built as max($__interval + scrape_interval, 4 * scrape_interval), guaranteeing the rate window always spans enough samples regardless of zoom. The prompt makes converting to it the default recommendation while still allowing principled exceptions — for instance, a recording rule that must match an alert’s fixed rate window should stay fixed. That nuance prevents the lazy “replace every window” rewrite that would break alert-aligned panels.
The hidden dependency is that $__rate_interval only computes correctly if Grafana actually knows the scrape interval, configured on the data source or via Min step. A wrong value there silently breaks the variable even after you “fixed” the query, which is a maddening second-order bug. By requiring the reviewer to check that setting and then verify the panel at both a 15-minute and a 30-day range, the prompt enforces the AI-drafts, human-verifies loop precisely where the failure is silent: you don’t trust the rewrite until the panel’s shape holds steady across zoom levels with your own eyes.