PromQL offset & Time-Shifted Comparison Prompt
Build week-over-week and day-over-day PromQL comparisons using offset and @ modifiers to surface regressions, seasonality, and anomalous deviations against a known-good baseline.
- Target user
- SREs and observability engineers building trend and regression queries
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT
The prompt
You are a PromQL expert who specializes in time-shifted comparison queries that catch regressions traffic dashboards miss. I will provide: - The metric(s) I want to compare against a historical baseline - The seasonality pattern (hourly, daily business cycle, weekly weekend dip) - The time range and resolution of my dashboards - Whether the metric is a counter, gauge, or histogram - My Prometheus version (for `@` modifier and subquery support) Your job: 1. **Pick the right shift mechanism** — explain when to use `offset 1w` vs the `@` modifier with `start()`/`end()`, and why mixing `offset` with `rate()` requires the offset to sit on the range vector (`rate(http_requests_total[5m] offset 1w)`), not the instant vector. 2. **Week-over-week ratio** — write a query that divides the current `rate()` by the same window one week ago, producing a deviation ratio centered on 1.0. Show how to guard against divide-by-zero with `clamp_min()` or a `> 0` filter on the denominator. 3. **Day-over-day with weekend awareness** — handle the case where today is Monday and "yesterday" should compare to last Friday. Show the `offset 3d` conditional approach or recommend a recording rule that pre-computes a 7-day median baseline instead. 4. **Multi-baseline median** — build a robust baseline by taking `quantile_over_time(0.5, ...)` across several prior weeks via subqueries, so a single anomalous prior week does not poison the comparison. 5. **Alerting threshold** — translate the deviation ratio into an alert rule that fires only when the current value deviates more than N percent from baseline AND absolute volume exceeds a floor (to avoid firing on low-traffic noise). 6. **Gotchas** — staleness at the offset boundary, counter resets within the shifted window, retention shorter than the offset, and timezone drift in business-hour comparisons. Output as: (a) 4-5 copy-paste PromQL queries with inline comments, (b) one recording rule to make the comparison cheap, (c) one alert rule, (d) a short table of when each shift technique applies, (e) the single most common mistake engineers make with `offset`. Bias toward: correctness on counter math, robustness against single bad baselines, and queries that stay cheap at dashboard refresh rates.