Postmortems for Near-Misses: Learning Before the Outage

The best incident to learn from is the one that didn’t happen. A near-miss — the deploy that would have taken down checkout if the canary hadn’t caught it, the disk that hit 98% before someone noticed — hands you all the lessons of an outage with none of the cost. Yet almost no team writes them up, because there’s no page, no angry customer, no forcing function. This guide is about capturing near-misses on purpose, before the free lesson becomes an expensive one.

Why near-misses are worth more than they seem

High-reliability industries — aviation, medicine, nuclear — figured this out decades ago: near-misses are leading indicators, and they’re abundant. For every actual outage there are usually several close calls where the system got within one failed safeguard of disaster. Studying those close calls lets you fix the underlying weakness while it’s still cheap and while nobody’s adrenaline is running. An outage teaches you the same lesson but bills you for it in downtime, customer trust, and a stressful night.

The catch is that near-misses are easy to rationalize away. “The canary caught it, so the system worked, so there’s nothing to review.” That’s exactly backwards. The canary caught it this time — but the fact that a table-locking migration reached the canary at all is the finding. A near-miss is a successful defense that reveals a failure that got remarkably close.

When to trigger a near-miss postmortem

You can’t review every non-event, so set criteria. Write up a near-miss when any of these is true:

A single remaining safeguard prevented an outage. One more failure and it would have been real.
You got lucky, not good. Recovery worked for a reason you don’t actually enforce (“the secondary happened to be in another AZ”).
A safeguard fired that you didn’t know you were relying on — or one fired that you thought was disabled.
A responder said “that was close” and meant it. Trust that instinct.
The same near-miss keeps happening. A recurring close call is an outage on a timer.

Make it explicitly safe — encouraged, even — to raise a near-miss. If people get grilled for surfacing one, they’ll stop surfacing them, and you lose your best leading indicator.

A lightweight near-miss template

Near-miss postmortems should be lighter than incident postmortems — the whole point is low friction, so people actually write them. Skip the heavy impact quantification and keep it focused on the gap that was exposed.

# Near-Miss: [Short name]

- **Date:** YYYY-MM-DD
- **Raised by:** [Name]
- **Severity if it had landed:** [What the outage would have been]

## What almost happened
Two or three sentences. The failure that nearly occurred and what it
would have affected.

## What stopped it
The safeguard, luck, or person that prevented the outage — and, honestly,
which of those it was.

## How close it got
How many safeguards were left? What would the next failure have been?

## The gap it revealed
The real finding. Why did the failure get this far? What weakness let it
approach production / customers at all?

## Action items
| # | Action item                      | Owner | Type    | Ticket   | Due   |
|---|----------------------------------|-------|---------|----------|-------|
| 1 | Block this class earlier in CI   | A. R. | Prevent | JIRA-201 | 07-25 |
| 2 | Enforce the thing we got lucky on| B. K. | Prevent | JIRA-202 | 07-30 |

The two sections that matter most are “how close it got” and “the gap it revealed.” Everything else is context.

Facilitating without the adrenaline

Near-miss reviews have a different feel than incident postmortems: no crisis, no defensiveness, more room to think. Use that.

Resist “no harm, no foul.” The reflex to shrug off a near-miss is strong precisely because nothing broke. Counter it by focusing the conversation on how close it got, not on the happy ending.
Convert luck into enforcement. Every “we got lucky because X” is an action item to make X guaranteed rather than lucky. That’s the highest-value output of a near-miss review.
Keep it fast. A near-miss review that takes as long as a full incident retro won’t get repeated. Fifteen minutes and a lightweight doc is the target.

Common mistakes

Only reviewing outages. If your postmortem process triggers only on customer impact, you’re ignoring your cheapest lessons.
Punishing the messenger. Grill someone for raising a near-miss and you’ll never hear about the next one.
Over-formalizing. Heavy process kills near-miss reporting. Keep the bar to write one very low.
Recording the save, not the gap. “The canary worked” is not the lesson. “A dangerous change reached the canary with no earlier check” is.

Wrapping up

Near-misses are outages that gave you a discount. Set clear criteria for when to write one up, make it safe and easy to raise them, and focus each review on how close it got and the gap it revealed — then convert every stroke of luck into an enforced safeguard. A team that studies its near-misses fixes weaknesses while they’re still cheap, and meets far fewer of them at 3am.