Paging Hygiene: Cutting Alert Noise Before the Next Incident

Every on-call engineer I’ve ever worked with can tell you about the alert they ignored because it “always fires and it’s always nothing” — right up until the one time it wasn’t. That’s the real cost of noisy paging. It isn’t the interrupted sleep, bad as that is. It’s that a stream of low-value pages trains people to ignore pages, and a team trained to ignore its alerts will eventually sleep through the one that mattered. Alert noise doesn’t just annoy your on-call; it quietly degrades your detection.

Paging hygiene is the ongoing practice of keeping your alerts high-signal, so that when a pager goes off, it means something and people respond to it. This is a practical framework for it: what makes an alert worth a page, how to route the rest, and a recurring ritual to keep the noise from creeping back.

The test: would you want to be woken for this?

Every paging alert should pass one question: if this fires at 3 AM, is there something a human needs to do about it right now? If the honest answer is no — it’s informational, it self-resolves, it can wait for morning, or there’s nothing to do anyway — then it should not page. It can log, dashboard, ticket, or notify a channel, but it should not wake a person.

This single test, applied ruthlessly, eliminates most alert noise. The reason teams accumulate noise is that they add a page every time something surprises them and never remove one, so the pager slowly fills with alerts that fail this test but nobody ever revisits. Hygiene is mostly the discipline of asking the question again, regularly, for every alert you have.

Tiers: not everything is a page

The core move in cutting noise is recognizing that “something happened” and “wake a human” are different tiers. Route notifications by what response they actually need:

ALERT TIERS

PAGE (wakes a human, 24/7)
  Customer impact happening or imminent, AND action needed now.
  → the ONLY things that should page.

TICKET / BUSINESS-HOURS
  Needs a human eventually, but not at 3 AM. Degradations with
  workarounds, slow-burning capacity trends, non-urgent errors.

NOTIFY (channel / dashboard, no page)
  Worth knowing, no action required. Informational events,
  successful auto-remediations, low-severity blips.

LOG ONLY
  Recorded for later analysis, nobody notified.

Most alerts that currently page belong in the lower tiers. Moving them down doesn’t lose information — it just stops the information from waking someone who can’t act on it.

Make alerts symptom-based, not cause-based

A huge source of noise is alerting on causes instead of symptoms. You don’t primarily care that CPU is at 90% or that one node is unhealthy — you care whether customers are affected. Alert on the symptom that maps to impact (error rate, latency past SLO, checkout success rate) and you get one meaningful page for a real problem. Alert on every underlying cause and you get a storm of pages for conditions that may not even be hurting anyone, plus a fresh page for each contributing factor during a real incident.

Symptom-based alerting also solves the alert storm problem: when something big breaks, cause-based alerts all fire at once and bury the signal. A responder facing forty simultaneous pages spends the first ten minutes just figuring out which one is the actual problem. Alert on symptoms, and one incident produces roughly one page.

De-dupe, group, and add dependencies

Beyond what you alert on, how the alerts flow matters:

Deduplicate: the same condition should produce one alert, not one per check interval. A flapping service shouldn’t send twenty pages in ten minutes.
Group related alerts: if a shared dependency is down, group the downstream alerts into one incident rather than paging separately for every affected service.
Encode dependencies: if the database is down, don’t also page for every service that depends on it — that’s noise pointing at a symptom you already know. Suppress the dependents when the root is already alerting.
Set sane thresholds and durations: require a condition to hold for a meaningful window before paging, so a two-second blip doesn’t wake anyone.

Every page should be actionable — link the runbook

An alert that fires with no guidance on what to do is half-noise even when it’s legitimate, because the responder burns time figuring out what it even means. Every paging alert should link to a runbook — or at minimum describe what it means, what to check, and what a first response looks like. If you find yourself unable to write a runbook for an alert because there’s nothing to do about it, that’s your answer: it shouldn’t be a page.

The recurring noise-cleanup ritual

Noise creeps back the moment you stop watching, so hygiene has to be a habit, not a one-time cleanup. Run this on a regular cadence — monthly, or every on-call rotation:

ALERT NOISE REVIEW (run every rotation)

[ ] Pull the last period's pages. Which fired most?
[ ] For each frequent alert: was it actionable? Real impact?
[ ] Kill or downgrade the ones that failed the 3 AM test.
[ ] Find alerts that fired and were ignored → fix or remove.
[ ] Find incidents that DIDN'T page but should have → add coverage.
[ ] Check for storms: did one incident cause many pages? Group them.
[ ] Confirm every remaining page has a runbook link.
[ ] Track pages-per-shift over time; it should trend DOWN.

The two most valuable lines are the ones that cut both ways: kill the alerts that fired and were ignored, and add coverage for the incidents that should have paged but didn’t. Hygiene isn’t only about less noise — it’s about the right signal. Track pages-per-shift as a health metric; a rotation that’s steadily quieting down is a rotation that trusts its pager.

Common mistakes

Adding alerts, never removing them. The pager fills with cruft nobody revisits. Make removal part of the ritual.
Alerting on causes, not symptoms. Produces storms and pages for things that may not hurt anyone. Alert on customer impact.
No tiers. Everything pages, so nothing feels urgent. Route non-urgent signals to tickets and channels.
Pages with no runbook. The responder wastes the critical minutes decoding the alert. Link the runbook or downgrade it.
Treating hygiene as one-time. Noise regrows. It’s a recurring ritual, not a project.
Silencing instead of fixing. A permanent silence on a noisy alert hides it and whatever it might legitimately catch. Fix the alert, don’t just mute it forever.

Where AI helps cut the noise

AI is good at exactly the pattern-finding that manual alert review is tedious at. Point a model at your paging history and have it cluster alerts — which fire together, which fire and get ignored, which correlate with real incidents versus which are pure noise — to hand you a ranked list of candidates to kill, downgrade, or group. During an incident, it can also correlate an alert storm into a probable single root, so the responder sees “these thirty alerts are likely one database issue” instead of thirty separate fires. The human decides what to change; the model surfaces the patterns hiding in months of paging data. The Incident Response tool can help correlate noisy alert bursts into a single assessment.

Wrapping up

Paging hygiene protects the thing you most need in a real incident: an on-call team that still trusts its pager. Apply the 3 AM test to every alert, tier the ones that don’t pass it down to tickets and channels, alert on customer-impacting symptoms rather than every cause, de-dupe and group so one incident makes roughly one page, and run a recurring review so the noise can’t creep back. A quiet pager isn’t a lucky pager — it’s a maintained one, and it’s the difference between catching the real incident and sleeping through it.

AI alert-clustering suggestions are advisory. Humans decide what pages and what doesn’t.