How to Declare an Incident (and When Not To)
Declaring an incident late costs time; too often cries wolf. A senior SRE on when to declare, how to do it cleanly, and when to hold back.
- #incident-response
- #sre
- #on-call
- #process
- #reliability
Two failure modes bookend every incident program. In the first, nobody wants to be dramatic, so a real outage smolders as a “let me just look into this quietly” for forty minutes before someone finally says the word “incident” — and by then the customer damage is done and the response starts cold. In the second, every blip becomes a Sev1 fire drill, on-call burns out, and people start ignoring the declarations because most of them turn out to be nothing.
The skill isn’t declaring incidents. It’s declaring the right ones, fast, and confidently not declaring the rest. After years of watching teams get this wrong in both directions, here’s how I think about the decision and how to make the declaration itself clean.
What “declaring” actually buys you
Declaring an incident is not an admission of failure. It’s flipping a switch that starts a defined process: a dedicated channel opens, a coordinator is named, a comms cadence begins, and the right people get pulled in with the right authority. The whole value of the word “incident” is that it triggers machinery — everyone knows what happens next without arguing about it.
That’s why hesitating to declare is so costly. Every minute you spend privately deciding whether this “counts” is a minute the machinery isn’t running. Declaring early and standing down quickly is almost always cheaper than declaring late.
When to declare: clear triggers
Write these into your runbook so nobody has to make the call from scratch at 2 AM. Declare an incident when any of these is true:
DECLARE AN INCIDENT IF ANY OF THESE HOLD:
[ ] Customers are affected right now (errors, downtime,
data issues) and it isn't self-resolving.
[ ] You'll need more than one person, or a team you have to
page, to resolve it.
[ ] The cause is unknown and impact is growing.
[ ] There's any data loss, corruption, or security concern.
[ ] It will breach an SLO / customer commitment if it continues.
[ ] You're not sure whether it's serious — declare and downgrade.
That last line matters most. “When in doubt, declare” is the right default if your culture makes standing down cheap and blameless. If declaring is a big scary event that gets you a talking-to, people will hesitate, and hesitation is the expensive failure mode. Make declaring routine and low-drama so the reflex is to flip the switch.
When not to declare
Declaring everything is its own disease. It trains people to ignore declarations and it burns out on-call. Hold back — handle it as normal work, a ticket, or a heads-up — when:
- There’s no customer impact and none is imminent. A failed nightly batch job you can rerun before anyone notices is a ticket, not an incident.
- A single person can resolve it in the normal course of work. If you don’t need to coordinate anyone, you don’t need the coordination machinery.
- It’s already self-resolved and stable. Note it, watch it, write it up if it’s likely to recur — but don’t spin up a war room for a blip that’s already gone.
- It’s a known, tracked, low-severity issue with a workaround. That’s backlog, not a 2 AM page.
The test is simple: does this need coordinated response, or does it need a fix? If one person fixing it in business hours covers it, it’s not an incident.
How to declare cleanly
A messy declaration is almost as bad as a late one. A clean one takes about a minute and looks like this:
🚨 DECLARING INCIDENT
What: Elevated error rate on checkout, customers seeing failed payments.
Impact: Appears to affect all checkout attempts, started ~14:03.
Severity: SEV2 (working — may raise).
Coordinating: [name] — I'm running point.
Channel: #inc-2026-checkout
Next update: 15 min.
Six lines. It names the impact, sets a working severity, and — critically — names who’s coordinating and where the response lives. Everyone who joins the channel now has the same picture without scrolling back, and nobody has to ask “who’s in charge here?”
Do it in a dedicated channel, not in the middle of your busy team channel where it’ll get buried under standup chatter. And declare it in a place your comms and stakeholders can see the fact of the declaration, even if the technical work happens elsewhere.
Who gets to declare
Everyone. This is the counterintuitive part teams resist. The person best positioned to notice something is wrong is usually the first responder or the engineer who got paged — not a manager who has to be looped in first. If declaring requires permission from someone senior, you’ve built a delay into the exact moment where delay hurts most.
Give every on-call engineer explicit authority to declare, and back it with a blameless promise: nobody is ever criticized for declaring an incident that turned out to be minor. The cost of an over-declaration is a few minutes of a few people’s time. The cost of an under-declaration is the outage running unmanaged. That trade is not close.
Common mistakes
- The silent investigation. One engineer quietly digs for forty minutes before declaring. By the time the team joins, the customer damage is done and everyone’s starting cold. If it might be an incident, declare while you investigate, not after.
- Declaration by committee. Waiting for a manager or a group to agree it “counts.” Give first responders the authority and skip the meeting.
- The permanent Sev1. Declaring high is right; never downgrading is not. Re-classify as you learn so your metrics and your responders’ stress both stay honest.
- Declaring without naming a coordinator. An incident with no named point person is just a crowd in a channel. Always say who’s running it.
- Never standing down. An incident that fizzles out without a clear “we’re done” leaves people unsure if they’re still on the hook. Close it explicitly.
Standing down is part of declaring
Declaring well includes knowing how to undeclare. Don’t call it resolved the instant the graph recovers — confirm the fix held long enough to trust it and that customer impact is actually gone. Then state it plainly: “Incident resolved as of [time], standing down, postmortem owner is [name].” That one sentence releases the responders, tells stakeholders it’s over, and assigns the follow-up. And if you declared and it turned out to be nothing? Say that too, warmly: “Confirmed no customer impact, standing down, good call checking.” Making the false-alarm stand-down a non-event is exactly what keeps people willing to declare next time.
Where AI helps the decision
The declare/don’t-declare call is a fast classification under uncertainty, which is where a model is a useful second opinion — never the decider. Paste the symptoms, the affected components, and any user-impact signal and ask whether this meets your declaration criteria and at what working severity, plus what additional signal would change the answer. That last part tells you what to go check. You still own the call; the model just helps you make it in seconds instead of minutes. The Incident Response tool turns raw symptoms into the structured assessment that makes this an easy yes-or-no.
Wrapping up
Declaring an incident is a routine operational act, not a confession. Declare when there’s real or imminent customer impact, when you need to coordinate people, or when you’re genuinely unsure — and hold back when one person can quietly fix it in the normal flow of work. Make the declaration clean and fast, give every responder the authority to make it, and make standing down blameless. Get those right and you’ll catch the real incidents early and stop crying wolf on the rest.
AI classification suggestions are advisory. The on-call engineer or incident commander owns the decision to declare.
Related
- Incident Severity Classification: A Practical SEV1-to-SEV4 Guide
- The Incident Commander Role Explained for Engineering Teams
- Designing Incident Escalation Policies That Actually Reach Someone
Get 500 Battle-Tested DevOps AI Prompts — Free
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.