🚨 Prometheus Error Guide: 'rate should only be used with counters' Non-Counter rate() Misuse

Exact Error Message

This is a PromQL info-level annotation, not a hard error: the query still returns a result, but Prometheus (2.45+/3.x) warns that you have applied a counter function to something that does not look like a counter. It appears in the Graph/Explore UI as a yellow notice and in the API response under infos:

PromQL info: metric might not be a counter, this could be a bug in the instrumentation: attempted to apply rate() to a metric that does not appear to be a counter

rate should only be used with counters and native histograms

Older versions emit no annotation at all — the only symptom is nonsensical output: large positive spikes, or rate() values that swing negative, which a true rate over a monotonic counter can never do.

{instance="web1:8080"}  -1.4e+06

What the Error Means

rate(), irate(), and increase() assume their input is a counter: a value that only ever goes up, resetting to zero on process restart. They compute per-second change and explicitly correct for resets by treating any decrease as a counter reset and adding the pre-reset value back.

When you feed a gauge (a value that legitimately goes up and down — temperature, queue depth, memory in use) into rate(), every legitimate decrease is misread as a counter reset. The reset-correction logic then fabricates huge jumps, and the result is meaningless. The annotation is Prometheus telling you the series did not look monotonic. The query is syntactically and type-valid (it parses and type-checks fine) — the problem is semantic: wrong function for the metric’s nature.

Common Causes

1. rate()/increase()/irate() applied to a gauge

The classic mistake. Gauges like node_memory_MemAvailable_bytes, node_filesystem_avail_bytes, or a queue-length gauge are not counters:

rate(node_memory_MemAvailable_bytes[5m])   # gauge — wrong

2. rate() applied to an already-rated metric

A recording rule or exporter that already exposes a per-second value (e.g. ..._per_second) is a gauge; rating it again is nonsense.

3. Misreading counter resets / restarts

Even on a real counter, very frequent restarts or a counter that was reset administratively can produce odd rate() output — though rate() is designed to handle normal restarts.

4. Using _sum without _count

For a Summary/Histogram, x_sum and x_count are both counters. Computing an average needs the ratio of their rates, not rate(x_sum) alone:

rate(http_request_duration_seconds_sum[5m])   # rate of total seconds — rarely what you want alone

5. Mixing metric types in one expression

Combining a counter’s rate with a gauge directly, without understanding each side’s semantics, yields numbers that look fine but mean nothing.

6. Summary / histogram bucket misuse

Applying rate() to the wrong histogram series, or to histogram_quantile’s input incorrectly. _bucket series are counters and rate correctly; the quantile itself is a gauge.

7. Info-typed / metadata metrics

*_info metrics (e.g. kube_pod_info) are constant 1 gauges carrying labels. rate() on them is always zero or meaningless.

How to Reproduce the Error

Apply rate() to a known gauge and inspect the annotation:

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=rate(node_memory_MemAvailable_bytes[5m])' \
  | jq '{result: (.data.result | length), infos: .infos}'

{
  "result": 3,
  "infos": [
    "PromQL info: metric might not be a counter ... (used with rate)"
  ]
}

promtool query instant http://localhost:9090 'rate(node_memory_MemAvailable_bytes[5m])'

Diagnostic Commands

Read the metric’s declared # TYPE straight from the exporter’s /metrics — this is the authoritative answer to “counter or gauge?”:

curl -s http://web1:8080/metrics | grep -E '^# TYPE (node_memory_MemAvailable_bytes|http_requests_total)'

# TYPE node_memory_MemAvailable_bytes gauge
# TYPE http_requests_total counter

Inspect raw values over time — a counter only ever rises (until a reset); if the raw series goes up and down, it is a gauge:

curl -s 'http://localhost:9090/api/v1/query_range' \
  --data-urlencode 'query=node_memory_MemAvailable_bytes' \
  --data-urlencode 'start=2026-06-27T13:00:00Z' \
  --data-urlencode 'end=2026-06-27T13:30:00Z' \
  --data-urlencode 'step=60s' \
  | jq -r '.data.result[0].values[] | .[1]'

Spot impossible rate values — a negative rate() over a real counter is impossible, so a negative result is itself a diagnosis:

curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=rate(suspect_metric[5m]) < 0' | jq '.data.result | length'

Compare deriv() (the gauge-appropriate function) on the same series to sanity-check direction:

promtool query instant http://localhost:9090 'deriv(node_memory_MemAvailable_bytes[5m])'

Step-by-Step Resolution

1. Confirm the metric type from # TYPE in /metrics. Counter → rate() is correct. Gauge → switch functions.

2. For counters, keep rate()/increase()/irate():

   rate(http_requests_total[5m])

3. For gauges, use a gauge function — deriv() for per-second slope, delta() for absolute change over the window, or *_over_time aggregates for stats:

   deriv(node_memory_MemAvailable_bytes[5m])     # per-second trend
   delta(node_filesystem_avail_bytes[1h])        # change over the last hour
   avg_over_time(queue_depth[5m])                 # average level
   predict_linear(node_filesystem_avail_bytes[1h], 4*3600) < 0   # will it hit zero?

4. For Summary/Histogram averages, divide the rate of _sum by the rate of _count (both counters):

   rate(http_request_duration_seconds_sum[5m])
     /
   rate(http_request_duration_seconds_count[5m])

5. For latency percentiles, rate the _bucket counters (they are counters) and feed histogram_quantile:

   histogram_quantile(0.95,
     sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

6. For *_info metrics, never rate them — join them for their labels with group_left instead.

7. Re-run and confirm the infos annotation is gone and the values make physical sense (no negatives for a rate).

Prevention and Best Practices

• Always check # TYPE before reaching for rate(). The function matters more than the metric name; a name ending in _total is a strong (but not guaranteed) counter signal.
• Map functions to types deliberately: counters → rate/increase/irate; gauges → deriv/delta/*_over_time/predict_linear; histograms → histogram_quantile over rate(..._bucket[...]).
• Treat any negative rate()/increase() value as a red flag that the input is not a counter, and investigate immediately.
• Keep instrumentation honest: name counters with _total, expose them as # TYPE counter, and never reuse a counter name for a gauge.
• Watch for newer Prometheus infos annotations in query responses and dashboards — they catch these mistakes for free if you surface them.
• The free incident assistant can read a metric’s type and rewrite rate() into the correct gauge or histogram expression; browse more under Prometheus and monitoring.

Related Errors

• Prometheus Error Guide: ‘parse error: unexpected’ — rate() with the wrong arity or a missing range selector fails at parse time, before this semantic annotation can fire.
• Prometheus Error Guide: ‘binary expression must contain only scalar and instant vector types’ — type errors that arise when mixing rated and non-rated operands.
• Prometheus Error Guide: ‘Empty query result’ / No data — a rate() window that is too short returns empty, a related but distinct symptom.

Frequently Asked Questions

How do I know if a metric is a counter or a gauge? Read the # TYPE line in the exporter’s /metrics output. # TYPE x counter means use rate(); # TYPE x gauge means use deriv()/delta()/*_over_time. The metric name is only a hint — # TYPE is authoritative.

Why is my rate() value negative? A true counter never decreases (except on reset, which rate() corrects for), so a negative rate means the input is a gauge that legitimately went down. rate() misread that drop as a reset and produced garbage. Switch to deriv() or delta().

What should I use instead of rate() for a gauge? deriv(gauge[5m]) for per-second slope, delta(gauge[1h]) for total change over a window, predict_linear(gauge[1h], 3600) to forecast, or avg_over_time/max_over_time for level statistics.

How do I compute an average from a histogram or summary? Divide the rate of the _sum counter by the rate of the _count counter: rate(x_sum[5m]) / rate(x_count[5m]). Both are counters, so rating them is correct; the ratio is the per-event average over the window.

Is the ‘might not be a counter’ message an error that fails my query? No. It is an info annotation. The query returns results and the HTTP status is success, but the values are almost certainly meaningless. Treat the annotation as a correctness warning and fix the function/metric pairing.