🔍 The $__rate_interval Trap: Why Grafana rate() Panels Lie When You Zoom

Someone zooms into a Grafana panel to inspect a spike, and the graph goes flat. Or empty. The service is fine, the data is there, but the line vanishes the moment the time range shrinks. Then they zoom back out and it reappears. This is one of the most common Grafana-on-Prometheus bugs, it generates a steady trickle of “is monitoring broken?” tickets, and it almost always comes down to a single wrong choice: using $__interval inside rate() where you should have used $__rate_interval.

The three interval variables

Grafana exposes a few time variables, and confusing them is the root cause:

• $__interval — the step Grafana computes from the selected range divided by panel width. It’s the time-per-pixel. Zoom in and it shrinks.
• $__range — the full selected range (e.g. 15m, 30d). Useful for _over_time aggregations, not for rate() windows.
• $__rate_interval — purpose-built for rate(). It’s defined as max($__interval + scrape_interval, 4 * scrape_interval), which guarantees the rate window always spans at least a few scrapes no matter how far you zoom.

That definition of $__rate_interval is the whole point. It can never shrink below roughly four scrape intervals, so rate() always has enough samples to compute a value.

Why $__interval breaks rate()

rate() needs at least two samples inside its window to compute a per-second rate. With a 15-second scrape interval, that means the window must be at least ~30 seconds, and realistically more to be stable. When you put $__interval in the window and zoom in, $__interval can drop below the scrape interval — the window now spans less than one scrape — and rate() returns nothing. No error. The panel just goes empty.

# Fragile: window shrinks when you zoom in, panel goes empty
rate(http_requests_total[$__interval])

# Correct: window never drops below ~4 scrape intervals
rate(http_requests_total[$__rate_interval])

A hardcoded window like rate(http_requests_total[5m]) doesn’t go empty, but it has the opposite problem: zoom out to 30 days and the 5-minute window smooths spikes into invisibility, so a real incident looks like nothing happened. $__rate_interval adapts to the zoom level, giving you a responsive window when zoomed in and a sensible one when zoomed out.

The hidden dependency: Grafana must know your scrape interval

Here’s the part that turns a “fixed” panel back into a broken one. $__rate_interval is computed using the scrape interval, and Grafana only knows that value if you tell it — via the Prometheus data source’s “Scrape interval” setting, or a panel’s Min step. If that’s wrong or default while your jobs scrape on a different cadence, $__rate_interval miscomputes and rate() misbehaves even though your query now uses the “right” variable. People fix the query, see no improvement, and conclude $__rate_interval is broken. It isn’t; the data source setting is.

Auditing a dashboard with AI

When you inherit a dashboard full of rate() panels, checking each one by hand is tedious. This is a clean review task for an assistant — paste the queries and the scrape interval and ask for a per-query verdict:

Here are my panel queries and the underlying scrape interval is 15s. For each rate()/increase() query, tell me whether it should use $__rate_interval, flag any using $__interval or a hardcoded window, and note any case where a fixed window is actually correct.

Panel A rate(http_requests_total[$__interval]) -> change to $__rate_interval; with a 15s scrape this goes empty when zoomed below ~30s. Panel B increase(errors_total[5m]) -> leave as-is only if it mirrors an alert’s fixed 5m window; otherwise switch to $__rate_interval. Panel C rate(...[$__rate_interval]) -> correct. Also confirm the data source “Scrape interval” is set to 15s, or $__rate_interval will miscompute regardless.

The model drafts the verdicts; you verify by loading each panel at a tight range (15m) and a wide one (30d) and confirming the shape holds with no gaps. That two-range check is the human-verifies half of the loop. The same review discipline shows up across the Grafana and PromQL prompts.

When a fixed window is actually right

$__rate_interval is the default, not a universal rule. There’s one important exception: a panel that must mirror an alert. If your alert fires on rate(errors_total[5m]) > 0.05, the dashboard panel that visualizes that alert should also use [5m], so what you see matches what pages you. Using $__rate_interval there would show a different window than the alert evaluates, and the graph and the page would disagree during an incident. Keep those panels fixed and deliberate.

The bottom line

If a Grafana rate() panel goes flat or empty when you zoom in, the window is shrinking below the scrape interval — switch it from $__interval to $__rate_interval and confirm the data source knows your scrape interval. Reserve fixed windows for panels that must match a specific alert. Then verify the fix by eye across a tight and a wide range, because this failure is silent: there’s no error, just a line that disappears and a teammate who thinks production went down. For a structured way to audit a whole dashboard, the $__rate_interval review prompt and the rate vs increase vs irate prompt cover the query side end to end.