🧪 Postmortem QA: Using AI to Catch Missing Sections and Unsupported Claims

A “finished” postmortem landed in my review queue with a confident root-cause section and not a single timestamp anywhere in the document. It asserted the outage lasted “around twenty minutes” with no evidence, blamed “a misconfiguration” without saying which one, and listed three action items that didn’t map to any cause described above them. The author wasn’t lazy—they were exhausted, and at the end of a long incident your own document looks complete because you remember the parts that aren’t on the page. Every postmortem has this blind spot. The fix is a QA pass, and it’s the most boring, most skippable, most valuable step in the whole process.

We QA code with linters and review. We almost never QA the postmortem, which is strange, because a postmortem with a hole in it ships bad conclusions straight into next quarter’s planning. AI is a near-perfect linter for this: it’s tireless, it doesn’t get defensive about its own writing, and it’s genuinely good at noticing structural gaps and claims with no support.

The three failures a postmortem QA pass catches

There are three classes of defect, and a human author is bad at all three on their own document.

Missing sections. No impact quantification, no detection-gap callout, no owner on the action items, a timeline that starts at “we noticed” instead of “it broke.” The author knows the missing info exists in their head, so the absence feels invisible.

Unsupported claims. “The root cause was a memory leak” with nothing in the timeline showing memory growth. “This affected a small number of users” with no number. Assertions that sound authoritative and have zero evidence beneath them. These are the most dangerous because they read as facts.

Unaddressed single points of failure. The postmortem describes how one component took everything down and then… never proposes removing the single point of failure. The incident demonstrated the SPOF and the action items quietly ignore it. This is the most expensive omission because it guarantees a sequel.

A QA prompt that interrogates the draft

I run this against the near-final draft. The framing matters: it’s a reviewer, not a rewriter. It points; I fix.

You are a strict postmortem reviewer. Do NOT rewrite the document.
Interrogate this draft and report problems only.

Check for:
1. MISSING SECTIONS — flag any of these that are absent or thin:
   summary, quantified impact, timeline (must include time-of-breakage,
   not just time-of-detection), root cause / contributing factors,
   detection gap, what went well, action items WITH owners.
2. UNSUPPORTED CLAIMS — list every assertion of fact (cause, impact,
   duration, scope) that has no evidence in the timeline or data.
   Quote the claim and say what evidence is missing.
3. SINGLE POINTS OF FAILURE — if the narrative describes a component
   whose failure caused broad impact, check whether an action item
   addresses it. If not, flag "SPOF demonstrated but not addressed".
4. ORPHANED ACTION ITEMS — any action item that doesn't trace to a
   cause or counterfactual described above it.
5. BLAME LANGUAGE — flag passive voice hiding decisions, or any
   phrasing that points at a person rather than a system.

Output a checklist with PASS/FAIL per item and a short fix suggestion
for each FAIL. Do not invent facts or fill gaps yourself.

The “do not fill gaps yourself” rule is load-bearing. If you let the model patch the holes, it’ll invent a plausible timestamp or a confident root cause, and now you’ve laundered a hallucination into an official document. You want it to find the hole, not paper over it.

The QA report you actually want

Postmortem QA — checklist

• Summary — PASS
• Quantified impact — FAIL: “a small number of users” has no figure. Need affected-user count and duration.
• Timeline — FAIL: earliest entry is “10:51 — alert fired.” No time-of-breakage. The detection gap can’t be computed.
• Root cause — PASS, but see unsupported claims.
• Detection gap — FAIL: not called out as its own finding.
• What went well — FAIL: absent.
• Action items with owners — FAIL: three items, zero owners.

Unsupported claims:

• “The root cause was a memory leak” — no memory metric in the timeline supports this. Add evidence or soften to “suspected.”
• “Resolved in twenty minutes” — no end timestamp. Compute from timeline.

SPOF: The narrative shows the single shared config service taking down all three regions. No action item proposes redundancy or graceful degradation. Flag: SPOF demonstrated but not addressed.

That SPOF flag is the one that earns the whole exercise. The author described the single point of failure vividly and then moved on, because in their head the cause was “the config service had a bug,” not “everything depends on one config service with no fallback.” The AI doesn’t have that mental shortcut, so it sees the structural problem plainly.

You still own every fix

The model finds the holes; a human fills them with real data and decides what’s a genuine SPOF worth fixing versus an acceptable risk you’re choosing to keep. Sometimes “SPOF demonstrated but not addressed” is the right answer—you’ve consciously decided redundancy isn’t worth the cost—but then the postmortem should say that explicitly, not stay silent. The QA pass forces the conversation; you make the call.

Two habits make this stick. Run the QA pass before the review meeting, not after, so the meeting discusses real findings instead of catching typos. And never let the QA pass become the author—its only job is to ask hard questions, and yours is to answer them with evidence. A postmortem that survives this interrogation is one people can trust, and trust is the only reason anyone reads the next one.

I keep this QA prompt with the rest of the incident set in the prompts library, and it pairs especially well with the impact and counterfactual work across the postmortems category. The template it’s checking against is in the blameless postmortem guide.

Lint your postmortem like you lint your code. The hole you can’t see is the one that ships.