🚨 Prometheus Error Guide: 'alert firing too often' — Stop Flapping Notification Spam

Overview

There is no single log line for this problem — the symptom is a stream of near-identical alert notifications for the same condition, arriving minutes (or seconds) apart. In Alertmanager it looks like the same alert cycling through firing and resolved states:

[FIRING:1] HighRequestLatency (api node-1:9100 warning)
[RESOLVED:1] HighRequestLatency (api node-1:9100 warning)
[FIRING:1] HighRequestLatency (api node-1:9100 warning)
[RESOLVED:1] HighRequestLatency (api node-1:9100 warning)

The underlying value hovers right at the alert threshold, so the rule evaluates true, then false, then true again, and every transition produces a page.

Symptoms

• The same alert pages repeatedly, sometimes several times per minute.
• Alerts flip between FIRING and RESOLVED without the real condition changing.
• The on-call channel is buried in notifications and engineers start ignoring them.
• The ALERTS metric shows the series appearing and disappearing (or toggling alertstate).
• Notifications keep arriving long after someone acknowledged the issue.

Common Root Causes

• No for: duration, so a single scrape over the threshold fires immediately.
• A for: that is too short relative to how noisy the metric is.
• A threshold sitting exactly where the metric normally oscillates, with no hysteresis.
• A jittery source metric (per-scrape rate spikes, GC pauses) that crosses the line on every evaluation.
• Alertmanager repeat_interval set very low, re-sending an already-firing alert too often.
• Poor grouping — group_interval too small, so each flap becomes a fresh notification instead of being batched.
• Missing inhibition, so a symptom alert keeps paging even when a parent cause alert is already firing.

Diagnostic Workflow

First confirm the flapping in PromQL. Look at how often the ALERTS series toggles for this alert:

count_over_time(ALERTS{alertname="HighRequestLatency", alertstate="firing"}[1h])

Then look at the raw signal the rule is built on. If it straddles the threshold, that is your flap source:

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="api"}[5m])))

Inspect the current rule. A missing or tiny for: is the most common cause:

# rules/api.yml
groups:
  - name: api
    rules:
      - alert: HighRequestLatency
        expr: histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="api"}[5m]))) > 0.5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "p95 latency high on {{ $labels.job }}"

Smooth the input so a single bad scrape cannot trip the rule. Averaging over a window is simple hysteresis:

avg_over_time(
  histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="api"}[5m]))
)[10m:]) > 0.5

Now tune Alertmanager so even a genuinely firing alert is not re-sent constantly:

# alertmanager.yml
route:
  receiver: on-call
  group_by: ['alertname', 'job']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = warning
      repeat_interval: 12h

Add inhibition so a high-severity parent silences its warning-level children:

inhibit_rules:
  - source_matchers: [severity = critical]
    target_matchers: [severity = warning]
    equal: ['job', 'instance']

Validate the config and inspect live alerts with amtool instead of clicking around the UI:

amtool check-config alertmanager.yml
amtool alert query alertname=HighRequestLatency --alertmanager.url=http://node-1:9093
amtool config routes test --config.file=alertmanager.yml severity=warning job=api

Example Root Cause Analysis

A team paged 40 times in one hour for HighRequestLatency. Querying count_over_time(ALERTS{alertname="HighRequestLatency"}[1h]) returned 40, confirming flapping rather than one sustained incident. Graphing the p95 expression showed it oscillating between 0.48s and 0.53s around the 0.5s threshold — normal jitter for that service.

Two problems compounded each other. The rule had for: 0s (omitted), so every scrape above 0.5s fired instantly, and Alertmanager’s repeat_interval was 5m, re-notifying constantly. The fix: add for: 10m so the condition must persist, wrap the expression in avg_over_time(...)[10m:] to smooth the jitter, and raise repeat_interval to 4h. After deploying, the alert fired once during a real latency regression and stayed quiet during normal oscillation.

Prevention Best Practices

• Always set a for: duration that exceeds the metric’s normal noise window; 5-15m suits most warning alerts.
• Place thresholds away from a metric’s steady-state band, and add hysteresis with avg_over_time or max_over_time for noisy signals.
• Keep repeat_interval at hours, not minutes, so acknowledged incidents do not re-page.
• Group related alerts with a sensible group_by and a group_interval of a few minutes to batch churn.
• Use inhibition rules so a root-cause alert suppresses its downstream symptoms.
• Review your busiest alerts monthly using ALERTS history and retire or retune the flappy ones. See /dashboard/monitoring-alerts/ for a rule-quality overview.

Quick Command Reference

# Validate rule and Alertmanager config
promtool check rules rules/api.yml
amtool check-config alertmanager.yml

# Inspect live/flapping alerts
amtool alert query alertname=HighRequestLatency --alertmanager.url=http://node-1:9093

# Test which route an alert takes
amtool config routes test --config.file=alertmanager.yml severity=warning job=api

# Count firing transitions in the last hour (flap detector)
count_over_time(ALERTS{alertname="HighRequestLatency", alertstate="firing"}[1h])

Conclusion

Notification spam is almost never caused by a real fault firing repeatedly — it is a metric grazing its threshold combined with an alert that reacts to every scrape. Add a for: duration, move or smooth the threshold with avg_over_time, and let Alertmanager batch and rate-limit with group_interval, repeat_interval, and inhibition. Verify with amtool and the ALERTS metric so you can prove the flapping is gone. More patterns live in the Prometheus stack guide.